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DISCLAIMER 


Certain commercial equipment and materials are identified in this report to specify adequately 
the technical aspects of the reported results. In no case does such identification imply 
recommendations or endorsement by the National Telecommunications and Infonnation 
Administration, nor does it imply that the material or equipment identified is the best available 
for this purpose. 
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ABBREVIATIONS/ACRONYMS/GLOSSARY 


Codec 

Coder 

Decoder 

df 

Encoder 

Fireground 

FM 

HATS 

IMBE™ 

LLR 

LRP 

MRT 

PASS 

PCM 

Pink Noise 
QFA 
QFB 
RE 

Vocoder 


Combination of an encoder and decoder in series (encoder/decoder). 

Same as "encoder". 

Device for translation of a signal from a digital representation into an analog 
format. For the purposes of this document, this is a device compatible with TIA- 
102.BABA [1]. 

Degrees of freedom for ANOVA or Newman-Keuls calculations. 

Device for converting an analog signal into a digital representation. For the 
purposes of this document, this is a device compatible with TIA-102.BABA[1]. 

The fire service term for an active fire scene. 

Frequency modulation. 

Head and torso simulator. 

Improved Multi-Band Excitation. A speech encoder which divides the audio 
spectrum into bands and generates speech model parameters based upon these 
spectral bands. In the speech decoder, the model parameters are used to 
synthesize speech for each of the spectral bands. 

Log likelihood ratio. 

Lip reference point of the HATS. 

Modified rhyme test as defined in [3], 

Personal alert safety system, which emits a signal in the event that the user 
becomes incapacitated or needs assistance. 

Pulse coded modulation, a logarithmically companded and 64 kb/s encoded 
representation of speech. 

Acoustic noise that has equal power per octave band as frequency increases. 
Full-rate baseline codec with no modulation (7200 bps). 

Enhanced full-rate codec with no modulation (7200 bps). 

Radio frequency. 

Voice encoder / decoder. 
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EXECUTIVE SUMMARY 


Project 25 (P25) is an initiative to develop a standardized digital radio and promote 
interoperability among digital land mobile radio (LMR) systems. The Association of Public 
Safety Communications Officials (APCO) and the Telecommunications Industry Association 
(TIA) cooperated to form this initiative in 1988. An integral part of the initiative, the digital 
voice coder/decoder pair (or vocoder for short) was selected in 1992, based on the results of 
several tests. This vocoder is named for the technology it uses, Improved Multi-Band Excitation 
(IMBE™), and was standardized by the TIA as TIA-102.BAB A. 

The performance of the IMBE vocoder has proven to be problematic in tactical fircground 
communications, where considerable background noise may be present. Several fire agencies 
have brought this problem to public attention (Boise, Fairfax, Littleton, and Phoenix Fire). The 
Digital Problem Working Group (DPWG) was formed by the International Association of Fire 
Chiefs (IAFC) in response to the problem, and is designed to provide input to testing procedures. 
The purpose of DPWG is to define the problem, identify potential solutions, and recommend best 
practices that could mitigate issues identified by first responders. 

One such testing procedure has been conducted. The Institute for Telecommunication Sciences 
in Boulder, Colorado developed the test plan in conjunction with the DPWG Testing 
Subcommittee. The goal of this experiment was to measure the intelligibility of communication 
systems operating in high acoustic noise environments typical of those encountered by 
firefighters. The three primary communication systems consist of: 

• 25 kHz analog FM radio pair (a.k.a. 25 kHz Analog) 

• Baseline full rate IMBE vocoder radio pair (a.k.a. P25 Full Rate) 

• Enhanced full rate IMBE vocoder radio pair (a.k.a. P25 Enhanced Full Rate) 

A modified rhyme test (MRT) was used to evaluate each communication system in each of nine 
environmental noise conditions (described below). The MRT method is more completely 
described in ANSI S3.2, and is required in the evaluation of self-contained breathing apparatus 
(SCBA) by the National Fire Protection Association (NFPA) 1981-2007 standard. 

In short, a basic MRT is a test where speakers are instructed to say “Please select the word,” 
followed by one of any number of words that “rhyme” on either the first or last consonant (e.g., 
one group of six words includes bed, led, fed, red, wed, and shed). These utterances are 
recorded, and later played back to test subjects, or listeners. The listeners are asked to select the 
word they heard at the end of the sentence. There are specific methods used to interpret this 
data, due to the situation where the listener has a limited set of possible answers to choose from, 
advance knowledge of the majority of the incoming sentence, as well as other constructs of the 
testing environment. 

In this test, the recorded utterances were processed using the vocoders on the aforementioned 
radio pairs. That processing took place while the radio pair was in the presence of one of the 
following nine environmental noise conditions: 
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• No background noise, no mask (or the Clean condition) 

• Fire truck pump panel, no mask 

• Mask with no background noise 

• Two Personal Alert Safety System (PASS) alanns, with mask 

• In-mask low-air alarm 

• Rotary saw cutting metal garage door, with mask 

• Chainsaw cutting wood, with mask 

• 2 1/2” hose with fog nozzle, with mask 

• Rotary saw cutting metal garage door, with amplified mask 

Those processed files were played back to listeners in a sound-isolated room. For the duration of 
the test, the room was filled with a field of pink noise. This test also evaluated a 12.5 kHz 
analog FM radio pair on three of the nine environmental noise conditions. The inclusion of this 
radio pair was to evaluate the intelligibility of this communication system, which meets an FCC 
mandate requiring narrowband devices. 

After 30 listeners participated in the test, the results were processed and some interesting 
conclusions can be drawn. The perfonnance of the 25 kHz Analog system was either statistically 
similar to or better than the P25 systems for all environments. Four of the nine environments 
were too difficult for intelligible communication using all tested systems (i.e., less than 10% 
intelligibility). The 12.5 kHz Analog system was statistically similar to the 25 kHz Analog 
system. 

This testing examined fire safety equipment used in conjunction with these systems. In the case 
of voice transmissions while using a mask, there was a significant degradation in intelligibility 
for the P25 vocoders. The PASS alarm, designed to augment safety of first responders, 
significantly degraded intelligibility of both P25 vocoders. The low air alann was effectively too 
difficult a noise environment for all of the tested communications systems, but it is worth noting 
that the 25 kHz Analog system and the P25 Enhanced Full Rate system preserved the noise 
characteristics of the low air alarm sufficiently well that a listener could determine what type of 
alarm was sounding. 

While this information may be useful to those planning to purchase and deploy new radio 
systems for their agencies, these results should not be the sole source of information. Other 
decision factors should include which agencies are involved, their current assets, their operating 
procedures, policies, and budget plans, spectrum availability in their locale, State 
Communications Interoperability Plans (SCIPs), and other, more specific situations that are 
beyond the scope of this document. 
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INTELLIGIBILITY OF SELECTED RADIO SYSTEMS IN THE 
PRESENCE OF FIREGROUND NOISE: TEST PLAN AND RESULTS 

David J Atkinson, Andrew A. Catellier 1 


This report describes an experiment conducted to measure the intelligibility of 
selected radio communication systems when those systems are employed in high- 
background-noise environments experienced by firefighters. The test plan for a 
Modified Rhyme Test (MRT) is detailed, including requirements for source 
material preparation and listening test conduct. Finally, the results of the test are 
presented, along with the data analysis. The results indicate that in some 
environments analog radios performed better than digital radios, and in some 
environments no radios performed well. This information should be considered 
whenever an agency is preparing to purchase and deploy a new communications 
system. 

Key words: intelligibility; Project 25; vocoder; modified rhyme test; noise; analog FM; land 
mobile radio; LMR; public safety; fire service 


1 INTRODUCTION 

Project 25 was initiated in 1998 as a cooperative agreement between the Association of Public 
Safety Communications Officials (APCO) and the Telecommunications Industry Association 
(TIA). The purpose of Project 25 was to develop a standardized digital radio to promote 
interoperability among digital land mobile radio (LMR) systems. One element of the Project 25 
suite, the vocoder, was selected in 1992. At that time, several tests were conducted by the 
Project 25 committees to ensure that the best available vocoder was selected. The selected 
vocoder, known as the Improved Multi-Band Excitation (IMBE™, a trademark of Digital Voice 
Systems Inc.), was standardized by TIA as TIA-102.BABA. 

As P25 networks were deployed, it became evident that certain noisy environments in which 
public safety must communicate are problematic for the vocoder. This has appeared most 
consistently in tactical fireground communications, and the issue has been raised to the national 
level by agencies such as Boise [Idaho] Fire, Fairfax [Virginia] Fire, Littleton [Colorado] Fire, 
and Phoenix [Arizona] Fire. In response to this issue, the International Association of Fire 
Chiefs (IAFC) created a Digital Problem Working Group (DPWG) to provide expert input into 
the testing to specifically identify the problem and potential solutions as well as to develop best 
practices that could mitigate some of the problematic issues in the communications environment. 

This document provides the test plan and results of the first test undertaken at the request of the 
DPWG. The test was designed to compare intelligibility of communication systems being used 
in fireground noise situations. The test plan was developed in conjunction with the Testing 
Subcommittee of the DPWG and the testing was conducted at the Institute for 
Telecommunication Sciences in Boulder, Colorado. This document describes the evaluation 
procedure used to characterize the response of digital voice coding technology to public safety 


1 The authors are with the Institute for Telecommunication Sciences, National Telecommunications and Information 
Administration, U.S. Department of Commerce, Boulder, CO 80305. 
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noise environments, as well as the results of that procedure. The report specifically focuses on 
those background noise environments experienced in the fireground. Understanding the 
characteristics of a digital voice coder is essential to enabling effective communication in the 
environment in which public safety personnel must operate. 

The evaluation procedure is designed to give a fair characterization of the communication 
system. This is accomplished through the examination of the system performance in a subjective 
listening test in which the relative performance among the enhancements is measured in a 
quantitative way. 

The purpose of the subjective listening test is to evaluate communication systems under a variety 
of operating conditions. The operating conditions were chosen to be representative of those 
experienced in a fireground environment. Only a limited number of operating conditions are 
tested. A test of all possible operating conditions would be too unwieldy to conduct. 

Section 4 describes the communication systems (and their respective vocoders) that are the 
subject of this test. Section 2 also discusses any background noise for the test conditions. 

Section 3 discusses the speech database used in the testing and comparison. Section 4 contains 
the details of the testing procedure as well as the speech database used for the testing. Section 5 
discusses the design of the listening tests themselves, including the analysis of results. Section 6 
describes the disclosure of the test results. Lastly, appendices are included which contain 
additional detail. 


1.1 Scope 

This document specifies the procedures to be employed to characterize the behavior of TIA 
102.BAB A compatible speech codecs in enviromnental noise conditions. The original baseline 
speech codec from 1992 is the IMBE described in TIA 102.BABA, Project 25 Vocoder 
Description. The IMBE speech codec is used to digitally encode the speech signal and provide 
forward error control for transmission at a data rate of 7200 b/s. The baseline speech codec is 
defined as the reference codec for the Project 25 MOS Conformance Test [2]. 

The testing of the codecs is performed using subjective listening tests that judge overall speech 
intelligibility [3], 

Where possible, this study compares systems incorporating the speech codecs with those using 
analog FM. The baseline speech codec is based upon the Digital Voice Systems Incorporated 
implementation of the IMBE algorithm on the VC 20 Project 25 hardware card, or the equivalent 
software version. The software version is of primary interest in this test. The experiment 
compares the coding mechanisms with various channel conditions as might occur on a land 
mobile radio channel. 


1.2 Overview 

Speech coders are tested by comparing their performance with a reference implementation such 
as analog FM. The test evaluates the intelligibility of communication systems using different 
vocoder technologies, including reference communication systems. The test results are then 
compared against the reference. There are four communication systems (with 4 different coding 
technologies) in this test. These mechanisms are listed below, and they are also described in 
Section 4. 
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System 1. 25 kHz analog FM radio pair. 


System 2. Radio pair implementing the baseline full rate IMBE vocoder. This is 
referred to as the “P25 Full Rate” throughout this report. 

System 3. Radio pair implementing the enhanced full rate. This is referred to as the 
“P25 Enhanced Full Rate” throughout this report. 

System 4. The experiment will also evaluate the viability of 12.5 kHz analog FM 
radio in three of the background noise conditions. 

The inclusion of the 12.5 kHz analog FM system provides an indication of whether or not that 
might be a viable alternative that meets the narrowbanding mandate of the FCC while still 
providing a required level of intelligibility to the communication system user. 

The intelligibility of each communication system is subjectively rated in each environmental 
noise condition specified. 

The intelligibility of a communication system can be difficult to quantify since it is a subjective 
issue, relying on humans to be able to discern words. This performance evaluation relies upon 
subjective testing using a panel of listeners who listen to speech passing through a system and 
attempt to understand what was spoken. Since discernment of listeners may vary, the results 
from a number of listeners are obtained and averaged to obtain an overall score. 

To evaluate the intelligibility, it is necessary to conduct an experiment in a controlled manner so 
that unintentional variation in the scoring is avoided. The purpose of the testing is to provide 
understanding of the behavior of communication systems in noisy environments and to 
determine differences in perfonnance among the different speech coding mechanisms. The 
confidence we have that any apparent differences in perfonnance are due to communication 
system effects and not random statistical variation depends upon how well we prevent 
differences from occurring in the testing. The statistical controls for the experiment and the 
analysis are given in Section 5. The listening test evaluates the communication systems under 
operating conditions, particularly different acoustic background noise conditions on the 
transmitting end of the communication path. The acoustic background noise conditions represent 
some noisy acoustic environments encountered by land mobile radio users in the fireground. 

This experiment uses the following noise environments on the transmitting end of the 
communication path: 

Environment 1. No background noise, no mask (referred to as the Clean condition) 
Environment 2. Fire truck pump panel, no mask 
Environment 3. Mask with no background noise 

Environment 4. Two Personal Alert Safety System (PASS) alarms, with mask 
Environment 5. In-mask low air alarm 

Environment 6. Rotary saw cutting metal garage door, with mask 
Environment 7. Chainsaw cutting wood, with mask 
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Environment 8. 214” hose with fog nozzle, with mask 

Environment 9. Rotary saw cutting metal garage door, with amplified mask 

Section 4.3 provides additional infonnation about the background noise. 

The system implementations are executed in hardware and recorded to generate test material. 

The test material consists of a series of computer files in WAV format, and the output files are in 
the same fonnat for use in the listening test. 

The overall plan of the test is outlined in Figure 1. The test begins with the source audio 
material. There are several conditions that the source material must satisfy, and these are 
covered in Section 3. The source audio material is then passed through the different 
communication systems, with different operating conditions, to produce numerous output audio 
files. This procedure is given in Section 4. The output audio files are then randomized in order 
to provide samples suitable for a listening test. The randomization step, together with the 
listening test and the analysis, is described in Section 5. The result of the test is then presented in 
Section 6, which describes a spreadsheet for this analysis. 



Figure 1. Test plan overview. 
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2 COMMUNICATION SYSTEMS AND NOISE ENVIRONMENTS 


This section describes the speech coding mechanisms evaluated in this test in a general way. 

The acoustic environments for the test are also described. 

2.1 Communication Systems 

There are four communication systems defined for this test. 

System 1. 25 kHz analog FM radio pair. This is the reference standard for fire 

agencies because it is the technology that they have been using since the 
mid-20 th century. 

System 2. Radio pair implementing the baseline full rate IMBE vocoder (QFA) , 

equivalent to the vocoder selected by Project 25 in 1992. This is described 
as a Full Rate vocoder. It operates at a gross bit rate of 7200 b/s, a net bit 
rate of 4400 b/s, with 2800 b/s of parity checks for channel error 
correction. This includes speech processing enhancements that have been 
incorporated by the radio manufacturer. This combination of a baseline 
vocoder with optional speech processing enhancements is referred to as 
the “P25 Full Rate” throughout this report. 

System 3. Radio pair implementing the enhanced full rate vocoder (QFB), 

interoperable with the baseline full rate vocoder. This vocoder operates at 
a gross bit rate of 7200 b/s, a net bit rate of 4400 b/s, with 2800 b/s of 
parity checks for channel error correction. It includes non-essential 
improvements developed since 1992 for improved audio quality built 
directly into the vocoder as well as any enhancements that may have been 
incorporated by the manufacturer. This combination of the enhanced 
vocoder with any manufacturer enhancements is referred to as the “P25 
Enhanced Full Rate” throughout this report. 

System 4. 12.5 kHz analog FM radio pair limited to three noise conditions. This is a 

system that could provide similar characteristics to 25 kHz Analog FM but 
still meet the impending 12.5 kHz narrowbanding requirements. 

The radio frequency (RF) communication path between transmit and receive units will be an 
ideal (cabled) path. 

2.2 Acoustic Environments 

The experiment tests the perfonnance of the communication systems (and their respective 
vocoders) with background noise mixed in with the speech at the transmitting radio for a specific 
signal to noise ratio. There are nine acoustic environments to be evaluated in this experiment: no 
background noise (no mask), fire truck pump panel (no mask), mask with no background noise, 
two PASS alarms (with mask), in-mask low air alarm, rotary saw cutting metal garage door (with 
mask), chainsaw cutting wood (with mask), a 214'” hose with a fog nozzle (with mask), and rotary 
saw cutting a metal garage door (with amplified mask). 

2 Appendix D contains a description of the notation used to identify codecs being tested in the TIA/Project 25 
community. In this case, the notation means: QPSK-c modulation (“Q”), full rate (“F”), baseline (“A”). 
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The noise conditions represent common acoustic noise environments where a user might be 
transmitting, and were chosen as representative by a panel of fire practitioners participating in 
the IAFC testing committee. The samples were taken as high-quality digital recordings made at 
an agency training facility. The signal-to-noise ratios (SNRs) are chosen to approximate sound 
level conditions that were measured in the application environment. 

The power spectral densities (PSDs) and spectrograms of the six background noises are given in 
Figure 2 to Figure 7. 

The PSDs in Figures 2-7 and the Acoustic Path Loss in Figure 8 were computed using Matlab™ 
mathematical analysis software. The computation method (Welch)[4], number of elements in the 
transform (Nfft), and the size and shape of the computational window (Hamming [5], 1024) were 
parameters provided to the Matlab algorithms. 

The spectrograms in Figures 2-7 are plots of frequency content (on the vertical axis) versus time 
(on the horizontal axis). 

Together the PSDs and spectrograms provide an indication that the noise environments cover a 
wide range of frequency characteristics, impulse characteristics, and amplitudes. 

The mask acoustic transfer characteristic is given in Figure 8. There is notable signal loss in the 
1,500 - 3,000 Hz range. This is significant because this range of frequencies is significant to 
intelligibility. 


2.3 Test Conditions 

The available systems combine with the noise environments to provide the array of conditions 
for the experiment. Table 1 shows which environments are used with which communication 
systems. Appendix A contains a full list of conditions for the experiment. 


Table 1. Combination of Noise Environments and Communication Systems 


Environment 

SNR 

(dB) 

25 kHz 
Ana FM 

P25 Full 
Rate 

P25 Enh 

Full Rate 

12.5 kHz 

Ana FM 

Clean 


X 

X 

X 

X 

Fire truck pump panel, no mask 

4 

X 

X 

X 


Mask with no noise 


X 

X 

X 

X 

Two PASS Alarms, mask 

-2 5 

X 

X 

X 


In-tnask low air alarm, mask 

15 J 

X 

X 

X 


Rotary saw cutting metal garage 
door, mask 

4 J 

X 

X 

X 

X 

Chainsaw cutting wood, mask 

5 J 

X 

X 

X 


2 V 2 ” fog nozzle, mask 

9 J 

X 

X 

X 


Rotary saw cutting metal garage 
door, amplified mask 

4 

X 

X 

X 



3 Plus attenuation of the signal due to the mask. This is approximately 9 dB. 
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Chainsaw Power 



Figure 2. PSD and spectrogram of a chainsaw cutting a wood roof. 
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Figure 3. PSD plot and spectrogram of circular saw cutting a metal garage door. 
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Figure 4. PSD plot and spectrogram of two PASS alarms sounding. 
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Figure 5. PSD plot and spectrogram of a 2Vi inch fire hose with fog nozzle. 
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Figure 6. PSD plot and spectrogram of a fire truck pump panel. 
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Figure 7. PSD plot and spectrogram of in-mask low air alarm. 
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Figure 8. Acoustic path loss through SCBA mask. 
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3 SOURCE SPEECH DATABASES 


The speech source material used for the test consists of spoken word lists from the Modified 
Rhyme Test (MRT) described in [3] (e.g., one group of six rhyming words includes bed, led, fed, 
red, wed, and shed). Source material used for the tests has a quiet acoustic background. In the 
experiment, acoustic background noise is added to the source material. The MRT is used for 
evaluation. These tests place several requirements on the speech database: 

1. Large numbers of words, 

2. Equalized presentation levels, 

3. Sentences of equivalent content and structure. 

The speech source material for the test has been recorded at ITS. This material is the MRT 
word list defined in [3] using the carrier sentence, “Please select the word ...” More specific 
information about the recording process for this source material is in Appendix C. 

3.1 Speech Database Requirements 

Each test condition uses the six lists of 50 words each defined in [3]. Each coding mechanism 
processes the same speech material under the same operating conditions. Sufficient source 
material is used to ensure listeners are not presented repeat material. To reduce order bias, the 
presentation order of the material to the listeners is randomized. 

The experiment utilizes speech recorded in a quiet acoustic noise environment. The testing 
includes a clean condition, and seven acoustic noise environments. For the experiments, there 
are three male talkers and three female talkers. 4 All talkers speak the same six lists of 50 words 
and are required by [3] to have little or no discernable accent. 

The speech material is equalized across all talkers for presentation level. The material is 
provided in WAV files in full audio bandwidth and dynamic range. There are no limitations on 
the speech material other than those imposed by the microphone and recording system. 

3.2 Speech Database Levels 

An important attribute of the speech database, especially for the encoding procedure, is the 
average power level of the speech material. The nominal power level for speech follows the 
recommendation defined in [1] and excerpted below. This recommendation is followed for each 
sentence. 

“It is recommended that the analog input gain be set such that the RMS speech level under 
nominal input conditions is 25 dB below the saturation point of the A-to-D converter. This level 
(-25 dBmO) is designed to provide sufficient margin to prevent the peaks of the speech waveform 
from being clipped by the A-to-D converter.” 

A further specification is given for the measurement method to determine the average speech 
power level. The varying nature of speech signal amplitude presents a difficulty for accurately 
measuring the level using an arbitrary method. This document uses the ITU-T Recommendation 
P.56 [6], method B to accurately measure the active speech level. 


4 [9] specifies an imbalanced talker pool of 4 male and 1 female talkers. To provide more gender-balanced results 
that could have wider applicability, a balanced talker pool of 3 male and 3 female talkers is used. 
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4 PRODUCTION OF PROCESSED FILES 


This section describes the test procedure to be followed to conduct the tests of the encoding 
technologies. These procedures have been designed to assist interested parties in reproducing the 
speech files for later scoring by a listening laboratory. 

4.1 Required Elements 

The production of the mixed speech and noise files requires the following elements. 

1. Head and torso simulators [7] [8] 

2. NC-35 sound attenuated chamber 

3. Representative SCBA mask with in-mask low-air alann 

4. Ability to produce environmental noise at appropriate level within the NC-35 
chamber 

5. Radios that can implement the coding technologies specified 

6. Recording and playback hardware and software 

4.2 Test Signal Preparation 

The test material generation procedure is most simply described as passing 300 sentences from 
each talker through nine noise environments and either three or four coding technologies for each 
speaker as identified in Table 1. This results in 54,000 processed sentences. 

For production of processed files, clean speech is played through a HATS speaking into a radio 
microphone while background noise is generated by loudspeakers in the attenuated chamber. 
Depending on the environment, there may be a SCBA mask installed on the HATS. 

For the conditions with background noise, it is important that the background noise be active 
before the push-to-talk is initiated to avoid potential false training of features of the codec (i.e., 
mislead the coder into thinking it is starting in a quiet environment), which may lead to longer- 
than-normal training times once the noise starts. 

Figure 9 shows the physical configuration for those conditions without a mask, and Figure 10 
shows the physical configuration for those conditions with a mask. The figures show the HATS 
in the middle of the attenuated chamber with loudspeakers around the perimeter. For those 
conditions without a mask, the transmitting microphone for the communication system will be 
positioned 5 cm (2 in.) directly in front of the lip reference point (LRP). For those conditions 
with a mask, the transmitting microphone of the communication system will be positioned 2.5 
cm (1 in.) directly in front of the voice transmission port on the mask. 

For single point background noise sources, the sound will be generated by the loudspeakers at 
the front of the room (in front of the HATS). For multipoint noise sources, all five speakers will 
be used to generate the noise. 
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For the purposes of this experiment, the artificial mouth of the HATS was equalized to flat ± 1 
dB in the band of 160 Hz to 10 kHz. Speech is played through the HATS at a level of 100 dBC. 5 
This is consistent with measurements made on behalf of TIA of users talking in a loud noise 
environment. 

For the receiving end, the speaker of the receiving radio will be positioned at the ear reference 
point (ERP). Volume on the receive radio will be set such that a 1011 Hz P25 test tone generates 
an acoustic signal of 85 dBA at 1” from the speaker grille. Recording will be done through the 
artificial ear of the second HATS in a “quiet” environment. This enables any listening 
environmental noise to be inserted during the actual listening experiment. 




Figure 10. Physical configuration for conditions with SCBA mask. 


5 dBC refers to a specific weighting function applied to a sound pressure level measurement. The “C” weighting 
function includes a broader spectrum of sound waves in the computation than the “A” weighting function. The “A” 
weighting function emphasizes the speech band and measurements using this function are measured in dBA. 


16 



4.3 Signal and Acoustic Noise Levels 


In order to construct the appropriate signal to noise levels for voice coder encoding, careful 
attention to the noise signal filtering and level adjustment of the signals is required. The 
definition for signal to noise ratio in dB for this test is given in (1). 

SNR = ActiveSpeeehLevel - 201og 10 ( NoiseLevel ) (1) 

Active speech level is computed according to [6]. The short-term power function of each noise is 
assumed to be much more stationary than speech, relieving the need for an activity/threshold 
detector. The computation for the noise levels follows a root mean square (RMS) algorithm that 
is scaled to the overload point so that dB values are negative or zero and is shown in (2). 


NoiseLevel 


1 


A 


fullscale 


1 


(=1 

N 


( 2 ) 


where Af u n sca ie is the fullscale amplitude of the signal, N is the number of samples, and x, is the 
value of the zth sample. 

The noise samples are scaled by the appropriate factor to obtain the target noise level for that 
specific test condition and summed together to create a noise condition file for voice coder input. 


17 



5 SUBJECTIVE EVALUATION OF INTELLIGIBILITY 


The subjective evaluation of intelligibility using the MRT is specified in [9] and described in [3]. 

Subjective testing involves the use of a number of listeners to attempt to interpret the words 
spoken through the communication system. The processed speech samples are obtained as 
described in Section 6. Because intelligibility is subject to individual abilities to interpret spoken 
words, reliable results are obtained when a number of listeners are used. For this assessment a 
total of 30 listeners are used in 6 groups of 5 listeners. To mitigate the effects of the order of 
presentation of the speech samples to the listeners, different presentation orders are used with 
each listener. 


5.1 Experiment Randomization 

The listening material consists of 6 lists of 50 words from each of 6 talkers (3 male, 3 female). 
This listening material is processed for each of the 30 test conditions yielding a total of 54,000 
test samples (50 words x 6 lists x 6 talkers x 30 test conditions). Each listener will score the 
intelligibility of 25 words from one list for one male and one female talker for each condition. 

To ensure that there are no effects of the order of presentation, the material used in the listening 
tests is presented to each listener in a different, randomized order. 

Randomization is constrained in the following ways. 

1. 25 words from one male and one female speaker for each test condition (combination 
of system and impairment) are presented exactly once to each listener such that no 
words are repeated (25x2x30 = 1500 samples per listener). 

2. Listeners are effectively paired in that consecutive odd/even numbered listeners will 
collectively hear all 50 words from one male and one female speaker for each test 
condition. 

3. Randomization is done in blocks of 60 test samples, such that one sample of each 
combination of codec, talker gender, and impairment is presented once, in each block, 
to each listener. There are 5 blocks in each listening session. This means that the 
listeners rate each combination of codec and impairment approximately equally in the 
beginning, middle, and end of the session. Each listener participates in 5 sessions 
during the experiment. 

4. The selection of the particular test samples for each block and their presentation order 
within a block is randomized. 

5. Consecutive talkers are always different. This is accomplished by alternating the 
gender of the talker. 


5.2 MRT Evaluation Laboratory 

The Evaluation Laboratory conforms where possible to the applicable sections of ITU-T 
Recommendation P.800 [10]. An Evaluation Laboratory is chosen to perform the evaluations. 
The Evaluation Laboratory is responsible for conducting the tests as described in Section 5.3 and 
the delivery of the results of the experiment as described in Section 5.4. 
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Prior to the start of the test the listeners participate in a practice session. During this practice 
session they are presented with 60 practice sentences which they score. The practice sentences 
consist of a block of material as described in Section 7.1.1 item 3, but taken from the larger 
corpus of speech material, excluding the samples that they score for the experimental sessions. 
After the practice session the listeners are asked if they understand what they are supposed to do. 
If there are any questions they are answered at that time. After that the formal test begins. The 
purpose of the practice session is to: (a) expose the listener to the range of audio quality of the 
test, and (b) to see if they understand what they are supposed to be doing. This is in accordance 
with P.800, clause B.4.6, “Instructions to Subjects.” 

The experimental results are presented, while the data from the practice sessions is discarded. 

The results from the listening test are then pennuted to undo the randomization described in 
Section 5.1 so that the results can be reported as described in Section 5.4. 

5.3 Intelligibility Testing 

The subjective evaluation consists of one experiment to detennine effects of background noise 
and talker variability. 

The listening test for the intelligibility test follows the Modified Rhyme Test (MRT) method [3], 
In this type of test, each listener listens to a sentence asking them to select a word from a 
prescribed list. The listeners’ ability to select the correct word is averaged across listeners and 
produces a percentage of intelligibility score. 

The MRT consists of the test conditions shown in Table 1. 

5.3.1 Conducting the Listening Test 

Presentation of speech material is made via high fidelity near-field monitor speaker at a distance 
of 1.5m from the listener. The delivery system is calibrated to deliver an average speech 
listening level of 75 - 85 dBA when measured at the listening position. The equivalent acoustic 
noise level of the delivery system does not exceed 35 dBA. 

Listeners are seated in a room, with an ambient pink noise level of 70 dBA as defined in Sections 
8.10.4.11 through 8.10.4.15 of [9], The exact configuration of the room and characterization of 
the noise is shown in Appendix E. 

That the listeners are practitioners notwithstanding, the listeners should be naive with respect to 
communication technology issues; that is, they are not experts in telephone design, digital voice 
encoding algorithms, and so on. The sample includes adults of mixed sex, age, and practitioner 
discipline. Persons have audiometrically nonnal hearing as defined in 8.10.4.3 of [9]. 

The test is conducted as described in Sections 8.10.5.1 and 8.10.5.2, with the exceptions that the 
carrier sentence will be “Please select the word [list word]” and that listeners will select the word 
on a touch screen. 

The administration of the experiment is as follows. The processed speech is presented to a panel 
of 30 listeners. The 30 listeners are segmented into five listening group sessions {ABCDEF} 
of 6 members each. Each listening group session contains 9,000 sentences out of the possible 
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pool of 54,000 6 sentences to evaluate. The 9,000 listening session sentences are further divided 
into 6 groups of 1,500 sentences for each listener of the group. Each listener will hear 5 lists of 
300 sentences. 

Before starting the test, the subjects are given the instructions in Figure 11. The instructions may 
be modified to allow for variations in laboratory data-gathering apparatus. 


[Before Training Session] 

Welcome and thank you for coming. 

This experiment is five sessions of approximately 20 minutes each. You will be able to take a 
break after each session, and we will have you take at least a 5 minute break after the second 
session. This experiment involves no risk or discomfort, and you are free to leave the 
experiment at any time for any reason, it will not be a problem for us. If you have any questions 
about the experiment, please feel free to ask them before the experiment starts. Your responses 
will be kept confidential, and will only be used as part of this experiment. 

This experiment uses the speakers in the room, so you will not be able to adjust the volume. 

The purpose of this experiment is to gather intelligibility information on systems that might be 
used for communications service between separate locations. You will be hearing a number of 
samples of speech reproduced in the speaker. Each sample will consist of the sentence “Please 
select the word X” spoken by male or female speakers. 

Please listen to the sentence, and then select the requested word from the list on the PDA. You 
may hear background noise in some of the samples. Please do your best to pick the requested 
word. 

Any questions? 

[Between Training Session and Session 1] 

Any problems during the training session? 

We will now do the first session of 300 samples. Any questions before we begin? 


Figure 11. Instructions to test subjects. 

5.4 Analysis and Reporting of Results 

The results are reported in a series of tables and figures in Section 6. The analysis and reporting 
is outlined in Sections 8.10.5 and 8.10.6 of [9], Averages are computed using the adjusted 
method recommended for closed set tests as described in Section 10.2 of [3]. An analysis of 


6 Note that only 45,000 of the 54,000 sentences will be used. This corresponds to 5 words from each block of 6 
words, defined in [3], for each talker for each condition. Selections from the extra words are used for training. 
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variance (ANOVA) will be computed to enable comparisons between the implementations. A 
more detailed description of the analysis follows. 

5.4.1 Intelligibility Scores 

The first result presented for each environment is a table of intelligibility scores. The 
intelligibility score represents the fraction of words that were selected correctly. This is the 
modified intelligibility score (RA) that accounts for potential correct guessing in the limited set 
of choices given to the listeners. This is either reported as a decimal number between 0.0 and 1.0 
or as a percentage. 

5.4.2 Background Information for ANOVA 

ANOVA and a multiple comparison test can assist in the detennination of whether there is a 
significant variation between the speech outputs of the three communication systems (and their 
respective vocoder technologies), and if so, which is better. A common multiple comparison test 
used in previous tests is the Tukey test. 

The data under analysis with ANOVA and Tukey consists of adjusted average intelligibility 
scores, RA (described in Section 10.2 of [3]), collected for 3 communication systems, 6 talkers, 8 
acoustic noise conditions, and 30 listeners. ANOVA compares the variance of the overall 
sample population with the variance within each sub-population, and if those variances exceed a 
value given by the Fisher F-distribution, then the null hypothesis is false. In this case, the null 
hypothesis is that communication systems do not make a difference in intelligibility. 

The hypothesis under test, H, is that the communication system (in particular the vocoder 
implemented by that communication system) affects the intelligibility, RA, as measured by the 
MRT. The hypothesis can be tested for each of the 8 acoustic noise conditions. If the hypothesis 
is true, it is also desired to know which communication system is better. 

For this test, the a = 0.01 for the F-distribution in the ANOVA for the given degrees of freedom, 
df, and sample size, n. 

5.4.3 Input to ANOVA 

A table is generated for each noise condition that includes the per-talker and per-communication- 
system average and standard deviation. 


Averages of means and standard deviations are computed for a sample size of 6 talkers and 30 
listeners, N sarap ie = 180. Representative calculations of mean and standard deviation for each 
noise condition, for a communication system is shown in Equation 3 and Equation 4, 
respectively. 


*»= 


2 »>'.£) 


N 


sample 


for vocoder v. Sum is over talkers, t=1..6, and listeners, L=l-30. 

S(v) = 




Z[(R A (v,t,L)-R A (v)Y 


N 


sample 


1 


( 3 ) 

( 4 ) 
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From the listening test description in 5.3.1, each listener will hear 25 words ('A of a 50-word list) 
for each talker for each condition. An RA value is computed for each talker for each condition. 
To achieve the required sample indicated in 5.3.1, 10 different listeners will hear different groups 
of 25 words for each talker and condition. This results in 10 RA values for each talker and 
condition. 

5.4.4 Example ANOVA Calculation and Multiple Comparison Test 

The analysis of data will be done in the statistical analysis package Minitab™. A sample Minitab 
report of the computation of the ANOVA and the Tukey test is shown below. Annotations are 
provided after each section of the report. 

One-way ANOVA: RA versus System 


Source DF SS MS F P 

System 3 0.31772 0.10591 13.40 0.000 

Error 236 1.86493 0.00790 

Total 239 2.18266 

S = 0.08889 R-Sq = 14.56% R-Sq(adj) = 13.47% 


The above represents the ANOVA results. The value of P is compared with the statistical 
significance of a=0.01. If P < a, statistically significant differences exist. 






Individual 99% CIs For Mean Based on 

Pooled StDev 

1 

60 

0.88080 

0.07830 

(-*-) 

2 

60 

0.82560 

0.08924 

(-*-) 

3 

60 

0.80000 

0.11184 

(-*-) 

4 

60 

0.88560 

0.07075 

( -*-) 

Pooled 

StDev = 0.08889 

0.800 0.840 0.880 0.920 


The above shows a rough graphic of the mean values and standard deviations for each of the four 
systems (Levels) in the test. The definition of each system is not important for this example. A 
pooled standard deviation value is also presented. 


Tukey 99% Simultaneous Confidence Intervals 
All Pairwise Comparisons among Levels of System 

Individual confidence level = 99.81% 


Corresponding to the a=0.01, the Tukey simultaneous confidence interval is set to 99%. This 
provides an individual comparison confidence interval of 99.81%. 


System = 1 subtracted from: 


System Lower Center Upper 

2 -0.10627 -0.05520 -0.00413 

3 -0.13187 -0.08080 -0.02973 

4 -0.04627 0.00480 0.05587 


System H-H-H-+ 
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2 ( -*-) 

3 ( -*-) 

4 ( -*-) 

+-+-+-+- 

-0.140 -0.070 0.000 0.070 

This first comparison is System 1 to the other systems. If the other systems’ values are all either 
above or below zero, then there is a statistically significant difference. If the other systems upper 
and lower values cross zero, then no statistically significant difference was detected. In this case, 
System 2 and System 3 are statistically significantly different from System 1. System 4 is not 
significantly different from System 1; this can also be stated as System 1 and System 4 are 
statistically similar. 

System = 2 subtracted from: 

System Lower Center Upper 

3 -0.07667 -0.02560 0.02547 

4 0.00893 0.06000 0.11107 


System H-H-H-H- 

3 ( - * - ) 

4 ( - * - ) 

+-+-+-+- 

-0.140 -0.070 0.000 0.070 


The second comparison is System 2 to System 3 and System 4. In this case System 2 and 
System 3 are statistically similar, but System 2 and System 4 are statistically significantly 
different. System 1 is not included in this graph because it was previously compared to System 2. 


System = 3 subtracted from: 


System 

Lower 

Center 

Upper 

+- 

-+- 

-+- 

-+- 

4 

0.03453 

0.08560 

0.13667 

-0.140 

-0.070 

0.000 

(-*- 

-+- 

0.070 


The third comparison is System 3 to System 4. In this case, System 3 and System 4 are 
statistically significantly different. System 1 and System 2 are not included in this graph because 
they were compared with System 3 in previous sections of this Minitab report. 

Once the comparisons are made, the infonnation can be synthesized into a more readable format 
as demonstrated in Table 2. A tabular result is presented in Section 6 for each environment in 
which the ANOVA detected a significant difference. Full Minitab reports for each environment 
are included in Appendix A. 

Table 2. Example Presentation of Tukey Results 

Tukey Multiple Comparison Results 

"YES" means significant difference 


System 

4 

3 

2 

1 

1 

NO 

YES 

YES 


2 

YES 

NO 



3 

YES 




4 
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6 DISCLOSURE OF TEST RESULTS 


This section describes the results of the experiment. One-way ANOVA results and Tukey 
comparisons were computed using the Minitab statistical analysis package. Input to the ANOVA 
consisted of 10 adjusted intelligibility scores (RA) per talker/noise environment/system. Each of 
those adjusted intelligibility scores were based on the intelligibility of 25 samples for that 
particular talker/environment/system combination that were presented to that particular listener. 

The RA listener scores for the experiment are tabulated in Appendix B. The analysis of the 
scores for the experiment is given in Section 6.1. 

6.1 Experiment Results Detail 

Experimental results are presented below for each noise environment in the test. For each noise 
environment, a table of the adjusted intelligibility scores (RA) and an analysis of the one-way 
ANOVA (p=0.01) is presented. If a statistically significant difference was detected by the 
ANOVA, a Tukey comparison analysis (a = 0.01) follows to reveal which elements are different, 
from a statistical standpoint. 

Finally, the results for the noise environment are presented in bar charts with statistically similar 
systems linked together by a horizontal red bar. If the red bar includes a dotted line, the systems 
crossed by the dotted portion of the line are not included in the statistical similarity. For example, 
in Figure 12, the top red bar indicates that the 25 kHz Analog FM and the 12.5 kHz Analog FM 
are statistically similar and the bottom horizontal red bar is used to indicate that the P25 Full 
Rate and P25 Enh Full Rate are statistically similar. 
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6.1.1 Results for Noise Environment 1 - “Clean” 


The “clean” noise transmission environment consisted of the ideal communications case with no 
background noise and no SCBA mask. The listening environment was as described in 5.3.1. 
Table 3 contains the intelligibility scores, ANOVA, and Tukey results for this environment. The 
detailed Minitab report is found in Appendix A. 

Figure 12 contains the bar chart of results for this environment. The solid portions of the top 
horizontal red bar in Figure 12 are used to indicate that the two analog systems are statistically 
similar in this environment. The bottom horizontal red bar is used to indicate that the two digital 
systems are statistically similar. Both of the analog systems are statistically better than the 
digital systems. 

Table 3. Intelligibility Scores, ANOVA Results, and Tukey Results for Environment 1 


Intelligibility Scores (RA) 


25 kHz ANA FM 

P25 Full Rate 

P25 Enh Full Rate 

12.5 kHz ANA FM 

0.881 

0.826 

0.800 

0.886 


One-way ANOVA: RA versus Condition 


Source 

DF 

SS 

MS 

F 

P 

Significant? 

Condition 

3 

0.31772 

0.10591 

13.4 

0.000 

YES 

Error 

236 

1.86493 

0.0079 




Total 

239 

2.18266 






S = 0.08889 R-Sq = 14.56% R-Sq(adj) = 13.47% 


Tukey Multiple Comparison Results 


System 

"YES" means 
significant difference 

4 3 2 1 

1-25 kHz ANA FM 

NO YES YES 

2 - P25 Full Rate 

YES NO 

3 - P25 Enh Full Rate 

YES 

4-12.5 kHz ANA FM 
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Intelligibility Scores for Environment 1 



Figure 12. Intelligibility scores and statistical equivalences for environment 1. 
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6.1.2 Results for Noise Environment 2 - Fire Truck Pump Panel 

The fire truck pump panel noise transmission environment consisted of the communications case 
with 4 dB SNR and no SCBA mask. The listening environment was as described in 5.3.1. Table 
4 contains the intelligibility scores, ANOVA, and Tukey results for this environment. The 
detailed Minitab report is found in Appendix A. 

Figure 13 contains the bar chart of results for this environment. The solid portions of the 
horizontal red bar in Figure 13 are used to indicate that the 25 kHz Analog FM and the P25 
Enhanced Full Rate are statistically equivalent in this environment. Both of these are statistically 
better than the P25 Full Rate. 

Table 4. Intelligibility Scores, ANOVA Results, and Tukey Results for Environment 2 


Intelligibility Scores (RA) 


25 kHz ANA FM 

P25 Full Rate 

P25 Enh Full Rate 

0.437 

0.341 

0.470 


One-way ANOVA: RA versus Condition 


Source 

DF 

SS 

MS F 

P 

Significant? 

System 

2 

0.5376 

0.2688 11.07 

0.000 

YES 

Error 

177 

4.3 

0.0243 



Total 

179 

4.8376 





S = 0.1559 R-Sq= 11.11% R-Sq(adj) = 10.11% 


Tukey Multiple Comparison Results 


"YES" means 
significant difference 


System 

3 

2 

1 

1-25 kHz ANA FM 

NO 

YES 


2 - P25 Full Rate 

YES 



3 - P25 Enh Full Rate 
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Intelligibility Scores for Environment 2 

i 
0.9 
0.8 
0.7 
0.6 
0.5 
0.4 
0.3 
0.2 
0.1 
0 


Figure 13. Intelligibility scores and statistical equivalences for environment 2. 
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6.1.3 Results for Noise Environment 3 - SCBA Mask 


This noise transmission environment consists of the communications case with a user wearing an 
SCBA mask with no additional background noise. The listening environment was as described in 
5.3.1. Table 5 contains the intelligibility scores, ANOVA, and Tukey results for this 
environment. The detailed Minitab report is found in Appendix A. 

Figure 14 contains the bar chart of results for this environment. The solid portions of the top 
horizontal red bar in Figure 14 are used to indicate that the two analog systems are statistically 
equivalent in this environment. The bottom horizontal red bar is used to indicate that the two 
digital systems are statistically equivalent. Both of the analog systems are statistically better 
than the digital systems. 

Table 5. Intelligibility Scores, ANOVA Results, and Tukey Results for Environment 3 


Intelligibility Scores (RA) 


25 kHz ANA FM 

P25 Full Rate 

P25 Enh Full Rate 

12.5 kHz ANA FM 

0.785 

0.522 

0.591 

0.798 


One-way ANOVA: RA versus Condition 


Source 

DF SS 

MS F 

P 

Significant? 

System 

3 3.47 

1.1567 73.16 

0.000 

YES 

Error 

236 3.7313 

0.0158 



Total 

239 7.2013 




0.1257 

R-Sq = 48.19% R-Sq(adj) = 47.53% 




Tukey Multiple Comparison Results 


System 

"YES" means 
significant difference 

4 3 2 1 

1-25 kHz ANA FM 

NO YES YES 

2 - P25 Full Rate 

YES NO 

3 - P25 Enh Full Rate 

YES 

4-12.5 kHz ANA FM 



29 




Intelligibility Scores for Environment 3 



Figure 14. Intelligibility scores and statistical equivalences for environment 3. 


30 














































6.1.4 Results for Noise Environment 4 - Low-Air Alarm 


The low air alarm noise transmission environment consisted of the communications case where a 
user would be wearing a mask and need to communicate while the low-air alarm was sounding. 
The listening environment was as described in 5.3.1. Table 6 contains the intelligibility scores, 
ANOVA, and Tukey results for this environment. The detailed Minitab report is found in 
Appendix A. 

Figure 15 contains the bar chart of results for this environment. The solid portions of the top 
horizontal red bar in Figure 15 are used to indicate that the 25 kHz Analog FM and P25 
Enhanced Full Rate systems are statistically equivalent in this environment. The bottom 
horizontal red bar is used to indicate that the two digital systems are statistically equivalent in 
this environment. The 25 kHz Analog FM performs significantly better than the P25 Full Rate. 

Table 6. Intelligibility Scores, ANOVA Results, and Tukey Results for Environment 4 


Intelligibility Scores (RA) 


25 kHz ANA FM 

P25 Full Rate 

P25 Enh Full Rate 

0.165 

0.058 

0.115 


One-way ANOVA: RA versus Condition 


Source 

DF 

SS 

MS F 

P 

Significant? 

System 

2 

0.3454 

0.1727 11.1 

0.000 

Yes 

Error 

177 

2.7534 

0.0156 



Total 

179 

3.0988 





Tukey Multiple Comparison Results 


"YES" means 
significant difference 


System 

3 

2 

1 

1 - 25 kHz ANA FM 

NO 

YES 


2 - P25 Full Rate 

NO 



3 - P25 Enh Full Rate 
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Intelligibility Scores for Environment 4 

i 
0.9 
0.8 
0.7 
0.6 
0.5 
0.4 
0.3 
0.2 
0.1 
0 

25 kHz ANA FM P25 Full Rate P25 Enh Full Rate 

Figure 15. Intelligibility scores and statistical equivalences for environment 4. 
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6.1.5 Results for Noise Environment 5 - PASS Alarms 


This noise transmission environment consisted of the communications case where a SCBA clad 
person would need to communicate upon finding two downed firefighters with their SCBA 
alarms sounding. The listening environment was as described in 5.3.1. Table 7 contains the 
intelligibility scores, ANOVA, and Tukey results for this environment. The detailed Minitab 
report is found in Appendix A. 

Figure 16 contains the bar chart of results for this environment. The horizontal red bar in Figure 
16 is used to indicate that the P25 Full Rate and the P25 Enhanced Full Rate are statistically 
equivalent in this environment. The 25 kHz Analog FM performed significantly better than 
either of the two digital systems. 

Table 7. Intelligibility Scores, ANOVA Results, and Tukey Results for Environment 5 


Intelligibility Scores (RA) 


25 kHz ANA FM 

P25 Full Rate 

P25 Enh Full Rate 

0.581 

0.152 

0.206 


One-way ANOVA: RA versus Condition 


Source 

DF 

SS 

MS F 

P 

Signficant? 

System 

2 

6.5503 

3.2752 178.71 

0.000 

YES 

Error 

177 

3.2438 

0.0183 



Total 

179 

9.7942 





S = 0.1354 R-Sq = 66.88% R-Sq(adj) = 66.51% 


Tukey Multiple Comparison Results 


"YES" means 
significant difference 


System 

3 

2 

1 

1-25 kHz ANA FM 

YES 

YES 


2 - P25 Full Rate 

NO 



3 - P25 Enh Full Rate 
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Figure 16. Intelligibility scores for environment 5. 


34 



























6.1.6 Results for Noise Environment 6 - Rotary Saw 

This noise environment represents the communications case where a person wearing a mask 
would need to communicate in the vicinity of someone operating a gas-powered rotary saw. The 
listening environment was as described in 5.3.1. Table 8 contains the intelligibility scores, 
ANOVA, and Tukey results for this environment. The detailed Minitab report is found in 
Appendix A. 

Figure 17 contains the bar chart of results for this environment. The horizontal red bar in Figure 
17 is used to indicate that the four systems perform in a statistically equivalent manner in this 
environment. 

Table 8. Intelligibility Scores, ANOVA Results, and Tukey Results for Environment 6 


Intelligibility Scores (RA) 


25 kHz ANAFM 

P25 Full Rate 

P25 Enh Full Rate 

12.5 kHz ANA FM 

0.046 

0.015 

0.005 

0.059 


One-way ANOVA: RA versus Condition 


Source 

DF 

SS 

MS F 

P 

Significant? 

System 

3 

0.11807 

0.03936 3.95 

0.009 

YES 

Error 

236 

2.3502 

0.00996 



Total 

239 

2.46827 




S = 0.09979 

R-Sq 

= 4.78% R-Sq(adj) = 3.57% 




Tukey Multiple Comparison Results 


System 

"YES" means 
significant difference 

4 3 2 1 

1-25 kHz ANA FM 

NO NO NO 

2 - P25 Full Rate 

NO NO 

3 - P25 Enh Full Rate 

NO 

4-12.5 kHz ANA FM 
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Intelligibility Scores for Environment 6 



0.4 

0.3 

0.2 

0.1 - 

0 -I - ^ - 1 - ^ - 1 - 1 - ^ - 

25 kHz ANA FM P25 Full Rate P25 Enh Full Rate 12.5 kHz ANA FM 

Figure 17. Intelligibility scores and statistical equivalences for environment 6. 
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6.1.7 Results for Noise Environment 7 - Chainsaw 


This noise transmission environment represents the communications case where a person 
wearing a mask needs to communicate near someone operating a chainsaw. The listening 
environment was as described in 5.3.1. Table 9 contains the intelligibility scores, ANOVA, and 
Tukey results for this environment. The detailed Minitab report is found in Appendix A. 

Figure 18 contains the bar chart of results for this environment. The top horizontal red bar in 
Figure 18 is used to indicate that the 25 kHz Analog FM system and the P25 Full Rate system 
are statistically equivalent. The bottom horizontal red bar is used to indicate that the P25 Full 
Rate and P25 Enhanced Full Rate systems are statistically equivalent in this environment. The 
25 kHz Analog FM performs significantly better than the P25 Enh Full Rate. 

Table 9. Intelligibility Scores, ANOVA Results, and Tukey Results for Environment 7 


Intelligibility Scores (RA) 


25 kHz ANA FM 

P25 Full Rate 

P25 Enh Full Rate 

0.064 

0.019 

-0.002 


One-way ANOVA: RA versus Condition 


Source 

DF 

SS 

MS F 

P 

Significant? 

System 

2 

0.1349 

0.0674 5.99 

0.003 

YES 

Error 

177 

1.9922 

0.0113 



Total 

179 

2.1271 




S = 0.1061 

R-Sq = 

6.34% R-Sq(adj) = 5.28% 




Tukey Multiple Comparison Results 


"YES" means 
significant difference 


System 

3 

2 

1 

1 - 25 kHz ANA FM 

YES 

NO 


2 - P25 Full Rate 

NO 



3 - P25 Enh Full Rate 
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Intelligibility Scores for Environment 7 



Figure 18. Intelligibility scores and statistical equivalences for environment 7. 
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6.1.8 Results for Noise Environment 8 - Fog Nozzle 

This noise transmission environment represents the communications case where someone 
wearing a mask would need to communicate near a fire hose operating with a fog nozzle. The 
listening environment was as described in 5.3.1. Table 10 contains the intelligibility scores and 
ANOVA results for this environment. The detailed Minitab report is found in Appendix A. 

Figure 19 contains the bar chart of results for this environment. The horizontal red bar in Figure 
19 indicates that there were no detectable differences between the systems. 

Table 10. Intelligibility Scores and ANOVA Results for Environment 8 


Intelligibility Scores (RA) 


25 kHz ANA FM 

P25 Full Rate 

P25 Enh Full Rate 

0.110 

0.068 

0.075 


One-way ANOVA: RA versus Condition 


Source 

DF 

SS 

MS F 

P 

Significant? 

System 

2 

0.0593 

0.0297 2.35 

0.098 

NO 

Error 

177 

2.233 

0.0126 



Total 

179 

2.2923 





Tukey analysis is not presented because the ANOVA did not detect a significant difference. 

Intelligibility Scores for Environment 8 

i 

0.9 

0.8 

0.7 

0.6 

0.5 

0.4 


0.3 

0.2 

0.1 



25 kHz ANA FM P25 Full Rate P25 Enh Full Rate 


Figure 19. Intelligibility scores and statistical equivalences for enviromnent 8. 
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6.1.9 Results for Noise Environment 9 - Amplified Mask 

This noise transmission environment represents the communications case where a person with a 
voice amplifier attached to their mask would need to communicate near a person operating a 
rotary saw. The listening environment was as described in 5.3.1. Table 11 contains the 
intelligibility scores, ANOVA, and Tukey results for this environment. The detailed Minitab 
report is found in Appendix A. 

Figure 20 contains the bar chart of results for this environment. The horizontal red bar in Figure 
20 is used to indicate that the 25 kHz Analog FM is statistically equivalent to the P25 Full Rate 
and P25 Enhanced Full Rate systems for this environment. The P25 Enhanced Full Rate system 
performs significantly better than the P25 Full Rate system. 

Table 11. Intelligibility Scores, ANOVA Results, and Tukey Results for Environment 9 


Intelligibility Scores (RA) 


25 kHz ANA FM 

P25 Full Rate 

P25 Enh Full Rate 

0.039 

0.015 

0.092 


One-way ANOVA: RA versus Condition 


Source 

DF 

SS 

MS F 

P 

Significant? 

System 

2 

0.1852 

0.0926 7.27 

0.001 

YES 

Error 

177 

2.2546 

0.0127 



Total 

179 

2.4398 





S = 0.1129 R-Sq = 7.59% R-Sq(adj) = 6.55% 

Tukey Multiple Comparison Results 


"YES" means 
significant difference 


System 

3 

2 

1 

1 - 25 kHz ANA FM 

NO 

NO 


2 - P25 Full Rate 

YES 



3 - P25 Enh Full Rate 
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Intelligibility Scores for Environment 9 

i -I- 

0.9 

0.8 


0.7 

0.6 

0.5 



Figure 20. Intelligibility scores and statistical equivalences for environment 9. 
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7 SUMMARY AND CONCLUSIONS 


Based on emerging concerns among fire service practitioners, this report describes an experiment 
that was conducted to measure intelligibility of radio systems in the presence of fireground noise. 
The test method selected to evaluate intelligibility was the Modified Rhyme Test. 

Four communication systems (25 kHz analog, 12.5 kHz analog, Project 25 Full Rate, and Project 
25 Enhanced Full Rate) were included in the test. Nine transmission environments (listed below) 
were also identified for the test. 

Environment 1. No background noise, no mask (referred to as the Clean condition) 
Environment 2. Fire truck pump panel, no mask 
Environment 3. Mask with no background noise 

Environment 4. Two Personal Alert Safety System (PASS) alarms, with mask 
Environment 5. In-mask low air alarm 

Environment 6. Rotary saw cutting metal garage door, with mask 

Environment 7. Chainsaw cutting wood, with mask 

Environment 8. 214” hose with fog nozzle, with mask 

Environment 9. Rotary saw cutting metal garage door, with amplified mask 

The report describes the method for generating the recordings of noisy speech data, the 
requirements of the listening enviromnent, and the process for presenting the recorded speech to 
listeners. 

The results of the subjective testing show that there are environments where analog radios had 
higher intelligibility than the digital radios, and also that there are environments where none of 
the systems perform very well. Observations included the following: 

• The MRT test seems to be significantly more sensitive (i.e., more able to 
discriminate) to differences in communication paths than MOS tests. This is 
evidenced by the much greater number of occurrences of statistically significant 
differences between the P25 Full Rate and P25 Enhanced Full Rate vocoder 
systems and between the analog and digital systems than has occurred in previous 
MOS-based tests such as [12] and [13]. 

• Without masks or background noise (i.e., Environment 1) all four communication 
systems effectively meet the NFPA 1981-2007 goal of 80% intelligibility. 

• Four of the nine noise environments were effectively too difficult for all tested 
communication systems, with all systems well below 10%. In future testing, 
either noise reduction technologies may be employed to evaluate these 
environments, or the environments may be adapted to provide more useful results. 
Another alternative might be context-based intelligibility testing. 


42 



• In environment 3 (SCBA mask with no additional background noise), the two 
analog systems effectively maintain the NFPA 1981-2007 goal of 80% 7 
intelligibility through a mask, while the digital systems were 52% for the P25 Full 
Rate vocoder system and 59% for the P25 Enhanced Full Rate vocoder system. 
This implies that there may need to be some additional considerations when using 
these digital communication systems in environments where a communicator 
must wear a SCBA mask for safety purposes. 

• In environment 2 (fire truck pump panel noise without an SCBA mask), both the 
analog system and the P25 Enhanced Full Rate system vocoder performed 
significantly better than the P25 Full Rate system. 

• The largest performance difference between the analog and digital systems was 
environment 5 (SCBA mask with two PASS alarms). 

• Analog FM outperfonned P25 Full Rate 5 of the 9 environments tested. In the 
remaining four environments, all radios had equivalent performance in three of 
them. In one environment, the performance of the P25 Full Rate system was 
equivalent to that of Analog FM. 

• Analog FM outperformed P25 Enhanced Full rate in 4 of the 9 environments 
tested. In the remaining 5 environments, all radios had equivalent performance in 
three of them. In two environments, the performance of the P25 Enhanced Full 
Rate system was equivalent to that of Analog FM. 

• In the case of P25 Full Rate and P25 Enhanced Full Rate, the two were 
statistically equivalent in 8 of 9 environments, with the P25 Enhanced Full Rate 
system performing better than the P25 Full Rate in 1 of the 9 environments. 

• In the three environments where 12.5 kHz Analog FM was included, it performed 
equivalent with 25 kHz Analog FM. This system should be evaluated further. 

• The testing indicated that there are significant communication challenges 
regarding SCBA and other fire safety equipment. In the case of a mask with no 
additional background noise at the transmission site, there was a notable 
degradation in intelligibility for the digital systems. In addition, some low air 
alarm designs and PASS alanns may interfere with communications intelligibility. 

The information contained in this report may be of value to those planning to purchase and 
deploy new radio communication systems in their agencies. The infonnation here should NOT 
be the only factor utilized in the purchasing process. Other factors should include agencies 
involved, current assets, departmental operating procedures and policies, budgeting plans, 
spectrum availability, State Communications Interoperability Plans (SCIPs), and specific 
situations faced by the purchasing agency that are not covered in this report. 


7 While the actual value of the 25 kHz Analog FM system was 78%, this was statistically equivalent to 80% as 
computed by the T-test defined in [9]. 


43 



8 ACKNOWLEDGEMENTS 


Funding for this effort was provided jointly by the Department of Homeland Security Office of 
Interoperability and Compatibiliy, the National Institute of Standards and Technology Office of 
Law Enforcement Standards, and the Federal Partnership for Interoperable Communications. 

The work was conducted at the Institute for Telecommunication Sciences under the supervision 
of J.R. Bratcher. 

The authors of this report wish to express gratitude to the following organizations, whose help 
was instrumental in the conduct of this experiment: 

• The City of Boise, Idaho and the Town of Plainfield, Indiana for providing opportunities 
to record high-quality public safety environmental noise. 

• The following communities/agencies for adjusting schedules and allowing their public 
safety personnel to travel to Boulder to participate in the experiment: 

o The City of Boise, Idaho 

o The County of Fairfax, Virginia 

o The Town of Plainfield, Indiana 

o The City of Littleton, Colorado 

o The City of Plainfield, Illinois 

o The City of Coeur d’Alene, Idaho 

o The City of Philadelphia, Pennsylvania 

o The City of Riverside, Ohio 

o The City of Englewood, Ohio 

o The City of Huber Heights, Ohio 

• The following companies who loaned equipment or provided support for the experiment: 

o Scott Health and Safety 
o Motorola 
o Draeger Safety 
o EF Johnson 

o International Safety Instruments 


44 



9 REFERENCES 


[1] Project 25 Vocoder Description, TIA-102.BABA. 

[2] Project 25 Mean Opinion Score Conformance Test, TIA-102.BABB. 

[3] ANSI S3.2, American National Standard Method for Measuring the Intelligibility of 
Speech Over Communication Systems, 1989. 

[4] A.V. Oppenheim, R.W. Schafer, Digital Signal Processing, Englewood Cliffs, NJ: 
Prentice Hall Inc., 1975. 

[5] F.J. Harris, “On the use of Windows for Harmonic Analysis with the Discrete Fourier 
Transform,” Proceedings of the IEEE, Vol. 66, No. 1, Jan. 1978. 

[6] ITU-T Recommendation P.56, “Objective measurement of active speech level.” 

[7] ITU-T Recommendation P.57, “Artificial Ears.” 

[8] ITU-T Recommendation P.58, “Head and Torso Simulator (HATS) for 
T elephonometry. ” 

[9] NFPA 1981 Standard on Open-Circuit Self-Contained Breathing Apparatus (SCBA) for 
Emergency Services, 2007 Edition. 

[10] ITU-T Recommendation P.800, “Methods for subjective determination of transmission 
quality.” 

[11] ITU-T Recommendation G.191, “Software Tools for Speech and Audio Coding 
Standardization.” 

[12] Project 25 Vocoder Evaluation Mean Opinion Score Test, TIA TSB 102.BABE. 

[13] Experiment 3 MOS Test Plan for Vocoder Technology for Project 25, TIA TSB 
102.BABF. 


45 



46 



APPENDIX A: CONDITION LABELS AND MINITAB REPORTS 


This appendix contains the representation of specific condition number to a communication 
system and background noise. Table A-l contains the list of condition numbers for this 
experiment. 


Table A-l. Condition Number, Communication System, and Noise Environment 


Condition 

Label 

Communication System 

Background Noise 

1 

25 kHz Analog FM 

No mask, no background noise 

2 

25 kHz Analog FM 

No mask, fire truck pump panel (4 dB S/N) 

3 

25 kHz Analog FM 

Mask, no background noise 

4 

25 kHz Analog FM 

Mask, in-mask low air alarm (15 dB S/N) 

5 

25 kHz Analog FM 

Mask, two PASS alarms (-2 dB S/N) 

6 

25 kHz Analog FM 

Mask, rotary saw (4 dB S/N) 

7 

25 kHz Analog FM 

Mask, chainsaw (5 dB S/N) 

8 

25 kHz Analog FM 

Mask, 2!4” hose with fog nozzle (9 dB S/N) 

9 

25 kHz Analog FM 

Amplified mask, rotary saw (4 dB S/N) 

10 

P25 Full Rate 

No mask, no background noise 

11 

P25 Full Rate 

No mask, fire truck pump panel (4 dB S/N) 

12 

P25 Full Rate 

Mask, no background noise 

13 

P25 Full Rate 

Mask, in-mask low air alarm (15 dB S/N) 

14 

P25 Full Rate 

Mask, two PASS alarms (-2 dB S/N) 

15 

P25 Full Rate 

Mask, rotary saw (4 dB S/N) 

16 

P25 Full Rate 

Mask, chainsaw (5 dB S/N) 

17 

P25 Full Rate 

Mask, 2!4” hose with fog nozzle (9 dB S/N) 

18 

P25 Full Rate 

Amplified mask, rotary saw (4 dB S/N) 

19 

P25 Enh Full Rate 

No mask, no background noise 

20 

P25 Enh Full Rate 

No mask, fire truck pump panel (4 dB S/N) 

21 

P25 Enh Full Rate 

Mask, no background noise 

22 

P25 Enh Full Rate 

Mask, in-mask low air alarm (15 dB S/N) 

23 

P25 Enh Full Rate 

Mask, two PASS alarms (-2 dB S/N) 

24 

P25 Enh Full Rate 

Mask, rotary saw (4 dB S/N) 

25 

P25 Enh Full Rate 

Mask, chainsaw (5 dB S/N) 

26 

P25 Enh Full Rate 

Mask, 2!4” hose with fog nozzle (9 dB S/N) 

27 

P25 Enh Full Rate 

Amplified mask, rotary saw (4 dB S/N) 

28 

12.5 kHz Analog FM 

No mask, no background noise 

29 

12.5 kHz Analog FM 

Mask, no background noise 

30 

12.5 kHz Analog FM 

Mask, rotary saw (4 dB S/N) 


Sections A. 1 through A.9 contain the full Minitab reports for Environments 1 through 9, 
respectively. For help interpreting the Minitab reports, see Section 5.4.4. 
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A.l Minitab Report for Environment 1 


One-way ANOVA: RA versus System 


Source 

DF 

SS 

System 

3 

0.31772 

Error 

236 

1.86493 

Total 

239 

2.18266 


MS F P 
0.10591 13.40 0.000 
0.00790 


S = 0.08889 R-Sq = 14.56% 


R-Sq(adj) = 13.47% 


Individual 99% CIs For Mean Based on 
Pooled StDev 


Level 

1 

N 

60 

Mean 

0.88080 

StDev 

0.07830 

+ 

-+-+-+ ■ 

(-*-) 

2 

60 

0.82560 

0.08924 

(- 

— *-) 

3 

60 

0.80000 

0.11184 

(-* — 

-) 

4 

60 

0.88560 

0.07075 


(-*-) 

-+-+-+ ■ 


0.800 0.840 0.880 0.920 


Pooled StDev = 0.08889 


Tukey 99% Simultaneous Confidence Intervals 
All Pairwise Comparisons among Levels of System 

Individual confidence level = 99.81% 


System = 1 subtracted from: 


System Lower Center 

2 -0.10627 -0.05520 

3 -0.13187 -0.08080 

4 -0.04627 0.00480 


Upper 

-0.00413 

-0.02973 

0.05587 


System 

2 

3 

4 


+ - + - + - +-- 

(- * - ) 

( - * - ) 

( - * -) 

+-+-+-+-- 

-0.140 -0.070 0.000 0.070 


2 subtracted from: 


System = 

System 

3 

4 


Lower 

-0.07667 

0.00893 


Center 

-0.02560 

0.06000 


Upper 
0.02547 
0.11107 


System H-H-H-H- 

3 ( - * - ) 

4 ( - * - ) 

+-+-+-+- 

-0.140 -0.070 0.000 0.070 


System = 3 subtracted from: 

System Lower Center Upper H-H 

4 0.03453 0.08560 0.13667 

-0.140 -0.070 


+ 


0.000 


0.070 


) 
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A.2 Minitab Report for Environment 2 
One-way ANOVA: RA versus System 


Source 

DF 

SS 

MS 

F P 

System 

2 

0.5376 

0.2688 

11.07 0.000 

Error 

177 

4.3000 

0.0243 


Total 

179 

4.8376 



S = 0. 

1559 

R-Sq = 

11.11% 

R-Sq(adj) =10 


Individual 99% CIs For Mean Based on 
Pooled StDev 

Level N Mean StDev --4-4-4-4- 

1 60 0.4368 0.1383 (-*-) 

2 60 0.3408 0.1815 (-*-) 

3 60 0.4696 0.1443 (-*-) 

0.300 0.360 0.420 0.480 

Pooled StDev = 0.1559 


Tukey 99% Simultaneous Confidence Intervals 
All Pairwise Comparisons among Levels of System 

Individual confidence level = 99.64% 


System = 1 subtracted from: 

System Lower Center Upper 

2 -0.1799 -0.0960 -0.0121 

3 -0.0511 0.0328 0.1167 


- + - + - + - +- 

(-*-) 

(-*-) 

- + - + - + - +- 

-0.12 0.00 0.12 0.24 


System = 2 subtracted from: 


System Lower Center Upper -4-4-4-+- 

3 0.0449 0.1288 0.2127 (-*-) 

- + - + - + - +- 

-0.12 0.00 0.12 0.24 
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A.3 Minitab Report for Environment 3 


One-way ANOVA: RA versus System 


Source 

DF 

SS 

MS 

F P 

System 

3 

3.4700 

1.1567 

73.16 0.000 

Error 

236 

3.7313 

0.0158 


Total 

239 

7.2013 



S = 0. 

1257 

R-Sq = 

48.19% 

R-Sq(adj) = 


Individual 99% CIs For Mean Based on 
Pooled StDev 


Level 

N 

Mean 

StDev —+-+-+-+- 

1 

60 

0.7848 

0.1010 (-*-) 

2 

60 

0.5216 

0.1600 (-*-) 

3 

60 

0.5912 

0.1353 (-*-) 

4 

60 

0.7984 

0.0955 (-*-) 


0.50 0.60 0.70 0.80 

Pooled StDev = 0.1257 


Tukey 99% Simultaneous Confidence Intervals 
All Pairwise Comparisons among Levels of System 

Individual confidence level = 99.81% 


System = 1 subtracted from: 


System 

2 

3 

4 


Lower 

-0.3354 

-0.2658 

-0.0586 


Center 

-0.2632 

-0.1936 

0.0136 


Upper 

-0.1910 

-0.1214 

0.0858 


( — *—) 


- 0.20 




( — *—) 

0.00 




0.20 


System = 2 subtracted from: 


System 

3 

4 


Lower 

-0.0026 

0.2046 


Center 

0.0696 

0.2768 


Upper 

0.1418 

0.3490 


- + - + - + - 

( — * —) 

- + - + - + - 

- 0.20 0.00 0.20 


System 

System 

4 


= 3 subtracted from: 

Lower Center Upper 
0.1350 0.2072 0.2794 



- 0.20 



0.00 


( —*—) 

0.20 




0.40 


) 

0.40 




0.40 
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A.4 Minitab Report for Environment 4 


One-way ANOVA: RA versus System 


Source 

DF 

SS 

MS 

F P 

System 

2 

0.3454 

0.1727 

11.10 0.000 

Error 

177 

2.7534 

0.0156 


Total 

179 

3.0988 



S = 0. 

1247 

R-Sq = 

11.15% 

R-Sq(adj) = 


Individual 99% CIs For Mean Based on 
Pooled StDev 


Level 

N 


Mean 


StDev 

1 

60 

0 . 

.1648 

0 

. 1450 

2 

60 

0 . 

. 0576 

0 

. 1086 

3 

60 

0 . 

. 1152 

0 

. 1177 


- +-+-+- 

0.050 0.100 0.150 0 


Pooled StDev = 0.1247 


Tukey 99% Simultaneous Confidence Intervals 
All Pairwise Comparisons among Levels of System 

Individual confidence level = 99.64% 


System = 1 subtracted from: 


System Lower Center Upper --1-H-H- 

2 -0.1743 -0.1072 -0.0401 (-*-) 

3 -0.1167 -0.0496 0.0175 (-*-) 

-0.160 -0.080 0.000 


System 

System 

3 


= 2 subtracted from: 

Lower Center Upper 
-0.0095 0.0576 0.1247 




-0.160 



-0.080 


(- 

0.000 


+ — 
-) 


+ — 

200 




0.080 


- ) 

0.080 
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A.5 Minitab Report for Environment 5 


One-way ANOVA: RA versus System 


Source 

DF 

SS 

MS 

F P 

System 

2 

6.5503 

3.2752 

178.71 0.000 

Error 

177 

3.2438 

0.0183 


Total 

179 

9.7942 



S = 0. 

1354 

R-Sq = 

66.88% 

R-Sq(adj) =66 


Individual 99% CIs For Mean Based on 
Pooled StDev 


Level 

1 

N 

60 

Mean 

0.5808 

0.1259 (-*--) 

2 

60 

0.1520 

0.1255 (--* — ) 

3 

60 

0.2056 

0.1529 (--* — ) 


0.15 0.30 0.45 0.60 


Pooled StDev = 0.1354 


Tukey 99% Simultaneous Confidence Intervals 
All Pairwise Comparisons among Levels of System 

Individual confidence level = 99.64% 


System = 1 subtracted from: 


System 

2 

3 


Lower Center 
-0.5017 -0.4288 
-0.4481 -0.3752 


Upper 

0.3559 

0.3023 


( 

-0.48 


) 

-0.32 


+ 


-0.16 


+ 


0.00 


System 

System 

3 


= 2 subtracted from: 

Lower Center Upper 
-0.0193 0.0536 0.1265 


- + 


-0.48 


-0.32 


+ 


-0.16 


+ 


( 


0.00 


) 
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A.6 Minitab Report for Environment 6 


One-way ANOVA: RA versus System 

Source DF SS MS F P 

System 3 0.11807 0.03936 3.95 0.009 

Error 236 2.35020 0.00996 

Total 239 2.46827 

S = 0.09979 R-Sq = 4.78% R-Sq(adj) = 3.57% 


Individual 99% CIs For Mean Based on 
Pooled StDev 


Level 

N 


Mean 


StDev - 

-+- 

-+- 

1 

60 

0. 

,04640 

0. 

.11125 

( — 

k 

2 

60 

0. 

, 01520 

0. 

.08636 

( -* — 

-) 

3 

60 

0. 

, 00480 

0. 

.09263 (-- 

k 

-) 

4 

60 

0. 

, 05920 

0. 

.10686 


( - 


- +-+-+-+- 

0.000 0.035 0.070 0.105 

Pooled StDev = 0.09979 


Tukey 99% Simultaneous Confidence Intervals 
All Pairwise Comparisons among Levels of System 

Individual confidence level = 99.81% 


System = 1 subtracted from: 


System 

2 

3 

Lower 

-0.08853 

-0.09893 

Center 

-0.03120 

-0.04160 

Upper 

0.02613 

0.01573 

(- 

(-*- 

-*-) 

-) 


-+ 

4 

System 

-0.04453 0.01280 

= 2 subtracted from: 

0.07013 

(- 

-0.060 

-k 

0.000 

-) 

0.060 

-+ 

0.120 

System 

Lower 

Center 

Upper 

_+ — - 

_+_ 

_+_ 

-+ 

3 

-0.06773 

-0.01040 

0.04693 

(- 


-) 


4 

System 

-0.01333 0.04400 

= 3 subtracted from: 

0.10133 

-0.060 

(- 

0.000 

k 

0.060 

—) 

-+ 

0.120 

System 

Lower 

Center 

Upper - 

-+- 

-+- 

-+- 

-+ 

4 

-0.00293 

0.05440 

0.11173 

-0.060 

(- 

0.000 

k 

0.060 

0.120 
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One-way ANOVA: RA versus System 


Source 

DF 

SS 

MS 

F P 

System 

2 

0.1349 

0.0674 

5.99 0.003 

Error 

177 

1.9922 

0.0113 


Total 

179 

2.1271 



S = 0. 

1061 

R-Sq = 

6.34% 

R-Sq(adj) = 5 


Individual 99% CIs For Mean Based on Pooled StDev 


Level 

1 

N 

60 

Mean 

0.0640 

StDev 

0.1181 

+ 

+ 

(- 

-+- 

— *-) 

2 

60 

0.0192 

0.0990 


( -*- 


-) 

3 

60 

-0.0016 

0.1002 

(- 

-0.035 

-k 

0.000 

0.035 

-+- 

0.070 


Pooled StDev = 0.1061 


Tukey 99% Simultaneous Confidence Intervals 
All Pairwise Comparisons among Levels of System 

Individual confidence level = 99.64% 


System = 1 subtracted from: 


System Lower Center Upper H-H-H-H 

2 -0.1019 -0.0448 0.0123 (-*-) 

3 -0.1227 -0.0656 -0.0085 (-*-) 

+ - +-+-+ — 

-0.120 -0.060 0.000 0.060 


System = 2 subtracted from: 


System Lower Center Upper H-H-H-H 

3 -0.0779 -0.0208 0.0363 (-*-) 

+ - + - + - +-- 

-0.120 -0.060 0.000 0.060 
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One-way ANOVA: RA versus System 


Source 

DF 

SS 

MS 

F 

P 

System 

2 

0.0593 

0.0297 

2.35 

0.098 

Error 

177 

2.2330 

0.0126 



Total 

179 

2.2923 





S = 0.1123 R-Sq = 2.59% R-Sq(adj) = 1.49% 

Individual 99% CIs For Mean Based on Pooled StDev 


Level 

1 

2 

3 

N 

60 

60 

60 

Mean 

0.1096 

0.0680 

0.0752 

StDev 

0.1187 

0.1020 

0.1155 

+ 


___ + 

-+- 


\ 


/ 




/ 









0.030 

0.060 

0.090 

0.120 


Pooled StDev = 0.1123 
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One-way ANOVA: RA versus System 


Source 

DF 

SS 

MS 

F P 

System 

2 

0.1852 

0.0926 

7.27 0.001 

Error 

177 

2.2546 

0.0127 


Total 

179 

2.4398 



S = 0. 

1129 

R-Sq = 

7.59% 

R-Sq(adj) = 6 


Level 

N 

Mean 

StDev 

Individual 99% 

Pooled StDev 

-+- 

CIs For Mean Based on 

1 

60 

0.0392 

0.1113 

( - 

— *-) 

2 

60 

0.0152 

0.0768 

(-*- 

-) 

3 

60 

0.0920 

0.1412 


(-*-) 


0.000 0.040 0.080 0.120 


Pooled StDev = 0.1129 


Tukey 99% Simultaneous Confidence Intervals 
All Pairwise Comparisons among Levels of System 

Individual confidence level = 99.64% 


System = 1 subtracted from: 


System Lower Center Upper H-H-H-H- 

2 -0.0848 -0.0240 0.0368 (-*-) 

3 -0.0080 0.0528 0.1136 (-*-) 

+ - + - +-+- 

-0.140 -0.070 0.000 0.070 


System = 2 subtracted from: 


System Lower Center Upper H-H-H-H- 

3 0.0160 0.0768 0.1376 (-*-) 

+ - + - + - +- 

-0.140 -0.070 0.000 0.070 
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APPENDIX B: LISTENER SCORES 


This appendix tabulates the scores collected from the listener panels. It contains the listener 
number, the talker number, the condition number, and the adjusted intelligibility score (RA). 


Table B-l. Scores Collected from Listener Panels 


Listener 

Talker 

Condition 

RA 

1 

FI 

4 

0.28 

1 

FI 

12 


1 

FI 

15 

0.04 

1 

FI 

16 


1 

FI 

18 

0.088 

1 

FI 

19 


1 

FI 

22 


1 

FI 

28 


1 

F3 

1 


1 

F3 

2 

0.52 

1 

F3 

5 


1 

F3 

6 


1 

F3 

7 


1 

F3 

8 


1 

F3 

10 


1 

F3 

13 


1 

F3 

14 


1 

F3 

23 


1 

F3 

25 


1 

F3 

26 


1 

F3 

27 


1 

F4 

3 

0.76 

1 

F4 

9 


1 

F4 

11 


1 

F4 

17 


1 

F4 

20 


1 

F4 

21 


1 

F4 

24 


1 

F4 

29 


1 

F4 

30 


1 

Ml 

4 


1 

Ml 

12 


1 

SB 

15 


1 

mm 

16 


1 

mm 

18 


1 

mm 

19 

0.76 

1 

mm 

22 


1 

mm 

28 


1 

SB 

1 


1 

M3 

2 


1 

M3 

5 


1 

M3 

6 


1 


7 

0.04 

1 

M3 

8 

0.04 

1 

SB 

10 


1 


13 


1 


14 

0.04 

1 


23 


1 


25 

0.04 

1 


26 


1 

M3 

27 

0.184 

1 

M4 

3 

0.76 

1 

M4 

9 

-0.008 

1 

M4 

11 

0.232 

1 

M4 

17 

-0.008 


Listener 

Talker 

Condition 

RA 

1 

M4 

20 

0.52 

1 

M4 

21 

0.76 

1 

M4 

24 


1 

M4 

29 


1 

M4 

30 


2 

FI 

4 


2 

FI 

12 


2 

FI 

15 


2 

FI 

16 


2 

FI 

18 


2 

FI 

19 

1 

2 

FI 

22 


2 

FI 

28 


2 

F3 

1 


2 

F3 

2 


2 

F3 

5 

0.76 

2 

F3 

6 


2 

F3 

7 


2 

F3 

8 

0.04 

2 

F3 

10 

0.76 

2 

F3 

13 


2 

F3 

14 

0.28 

2 

F3 

23 


2 

F3 

25 


2 

F3 

26 

0.04 

2 

F3 

27 


2 

F4 

3 


2 

F4 

9 


2 

F4 

11 


2 

F4 

17 


2 

F4 

20 


2 

F4 

21 


2 

F4 

24 


2 

F4 

29 


2 

F4 

30 


2 

Ml 

4 


2 

Ml 

12 


2 

Ml 

15 


2 

Ml 

16 


2 

Ml 

18 

0.04 

2 

Ml 

19 

1 

2 

SBi 

22 

0.28 

2 

SB 

28 


2 

SB 

1 


2 

M3 

2 


2 

M3 

5 


2 

M3 

6 

0.04 

2 

M3 

7 

0.04 

2 

M3 

8 


2 

M3 

10 


2 

M3 

13 

0.088 

2 

M3 

14 

0.088 

2 

M3 

23 

0.04 

2 

M3 

25 

0.04 

2 

M3 

26 

0.136 


Listener 

Talker 

Condition 

RA 

2 

M3 

27 

-0.008 

2 


3 


2 


9 


2 


11 


2 


17 


2 


20 


2 


21 


2 


24 


2 


29 


2 


30 

0.04 

3 

FI 

3 


3 

FI 

8 

0.04 

3 

FI 

9 


3 

FI 

10 


3 

FI 

17 


3 

FI 

21 


3 

FI 

25 


3 

FI 

26 


3 

FI 

27 


3 

FI 

30 


3 

F3 

11 


3 

F3 

12 


3 

F3 

20 


3 

F3 

24 


3 

F3 

28 


3 

F3 

29 


3 

F4 

1 


3 

F4 

2 


3 

F4 

4 


3 

F4 

5 


3 

F4 

6 


3 

F4 

7 

0.04 

3 

F4 

13 


3 

F4 

14 

0.28 

3 

F4 

15 


3 

F4 

16 


3 

F4 

18 


3 

F4 

19 


3 

F4 

22 


3 

F4 

23 


3 

Ml 

3 


3 


8 


3 


9 


3 


10 


3 


17 


3 


21 


3 


25 


3 


26 

0.04 

3 


27 


3 


30 


3 

M3 

11 

0.04 

3 

M3 

12 

0.424 

3 

M3 

20 

0.424 

3 

M3 

24 

0.04 

3 

M3 

28 

0.904 
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Listener 

Talker 

Condition 

RA 

13 

Ml 

1 

1 

13 

Ml 

4 

0.136 

13 

Ml 

11 

0.184 

13 

Ml 

18 

-0.008 

13 

Ml 

21 

0.568 

13 

Ml 

22 

0.376 

13 

Ml 

23 

0.04 

13 

Ml 

25 

-0.056 

13 

Ml 

28 

0.856 

13 

M3 

2 

0.424 

13 

M3 

3 

0.712 

13 

M3 

7 

-0.008 

13 

M3 

8 

-0.056 

13 

M3 

14 

-0.056 

13 

M3 

15 

0.136 

13 

M3 

17 

0.28 

13 

M3 

26 

-0.104 

13 

M3 

29 

0.808 

13 

M3 

30 

-0.008 

13 

M4 

5 

0.376 

13 

M4 

6 

-0.104 

13 

M4 

9 

-0.104 

13 

M4 

10 

0.952 

13 

M4 

12 

0.664 

13 

M4 

13 

0.088 

13 

M4 

16 

0.04 

13 

M4 

19 

0.76 

13 

M4 

20 

0.328 

13 

M4 

24 

-0.008 

13 

M4 

27 

0.088 

14 

FI 

1 

0.856 

14 

FI 

4 

0.472 

14 

FI 

11 

0.616 

14 

FI 

18 

0.088 

14 

FI 

21 

0.616 

14 

FI 

22 

0.184 

14 

FI 

23 

0.424 

14 

FI 

25 

0.088 

14 

FI 

28 

0.952 

14 

F3 

2 

0.328 

14 

F3 

3 

0.808 

14 

F3 

7 

0.04 

14 

F3 

8 

-0.056 

14 

F3 

14 

0.28 

14 

F3 

15 

-0.2 

14 

F3 

17 

0.136 

14 

F3 

26 

0.136 

14 

F3 

29 

0.76 

14 

F3 

30 

0.136 

14 

F4 

5 

0.712 

14 

F4 

6 

-0.008 

14 

F4 

9 

0.088 

14 

F4 

10 

0.856 

14 

F4 

12 

0.472 

14 

F4 

13 

0.232 

14 

F4 

16 

-0.056 

14 

F4 

19 

0.808 

14 

F4 

20 

0.424 

14 

F4 

24 

-0.056 

14 

F4 

27 

0.088 

14 

Ml 

1 

0.904 

14 

Ml 

4 

0.28 

14 

Ml 

11 

0.28 

14 

Ml 

18 

0.088 

14 

Ml 

21 

0.616 


Listener 

Talker 

Condition 


14 

Ml 

22 


14 

Ml 

23 


14 

Ml 

25 


14 

Ml 

28 


14 

M3 

2 


14 

M3 

3 


14 

M3 

7 


14 

M3 

8 

0.088 

14 

M3 

14 


14 

M3 

15 

0.088 

14 

M3 

17 


14 

M3 

26 


14 

M3 

29 


14 

M3 

30 


14 

M4 

5 


14 

M4 

6 


14 

M4 

9 


14 

M4 

10 


14 

M4 

12 


14 


13 


14 


16 


14 


19 


14 


20 


14 


24 

0.04 

14 


27 


15 

FI 

5 

0.52 

15 

FI 

7 


15 

FI 

12 


15 

FI 

13 


15 

FI 

15 


15 

FI 

19 


15 

FI 

20 


15 

FI 

24 

0.04 

15 

FI 

26 


15 

FI 

27 


15 

FI 

29 


15 

FI 

30 

0.28 

15 

F3 

4 


15 

F3 

6 


15 

F3 

9 


15 

F3 

10 


15 

F3 

11 


15 

F3 

16 

0.04 

15 

F3 

18 


15 

F3 

25 


15 

F3 

28 


15 

F4 

1 

1 

15 

F4 

2 


15 

F4 

3 


15 

F4 

8 


15 

F4 

14 


15 

F4 

17 


15 

F4 

21 


15 

F4 

22 


15 

F4 

23 


15 

Ml 

5 


15 

Ml 

7 


15 

Ml 

12 


15 

Ml 

13 


15 

Ml 

15 


15 

Ml 

19 


15 

Ml 

20 


15 

Ml 

24 


15 

Ml 

26 

-0.008 

15 

Ml 

27 

0.184 


Listener 

Talker 

Condition 

RA 

15 

— 

29 


15 

msm 

30 


15 


4 


15 


6 


15 


9 


15 


10 


15 


11 


15 


16 


15 


18 


15 


25 


15 


28 


15 


1 


15 


2 


15 


3 

0.76 

15 


8 


15 

M4 

14 


15 

M4 

17 

0.04 

15 

M4 

21 


15 

M4 

22 


15 


23 

0.28 

16 

FI 

5 


16 

FI 

7 


16 

FI 

12 

0.76 

16 

FI 

13 


16 

FI 

15 


16 

FI 

19 

1 

16 

FI 

20 


16 

FI 

24 


16 

FI 

26 


16 

FI 

27 

0.28 

16 

FI 

29 


16 

FI 

30 


16 

F3 

4 


16 

F3 

6 


16 

F3 

9 


16 

F3 

10 

0.76 

16 

F3 

11 


16 

F3 

16 


16 

F3 

18 

0.04 

16 

F3 

25 


16 

F3 

28 


16 

F4 

1 


16 

F4 

2 

0.52 

16 

F4 

3 


16 

F4 

8 


16 

F4 

14 


16 

F4 

17 


16 

F4 

21 


16 

F4 

22 

0.28 

16 

F4 

23 


16 


5 


16 


7 


16 


12 


16 


13 


16 

Ml 

15 


16 

Ml 

19 


16 

Ml 

20 


16 

Ml 

24 

0.04 

16 

Ml 

26 


16 

Ml 

27 


16 

Ml 

29 

0.76 

16 

Ml 

30 


16 

M3 

4 

0.04 

16 

M3 

6 

-0.008 

16 

M3 

9 

0.04 
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Listener 

Talker 

Condition 

RA 

23 

F3 

14 

0.184 

23 

F3 

15 

-0.008 

23 

F3 

16 

0.04 

23 

F3 

20 

0.424 

23 

F3 

22 

0.088 

23 

F3 

24 

-0.104 

23 

F3 

30 

0.088 

23 

F4 

1 

0.808 

23 

F4 

4 

0.088 

23 

F4 

8 

0.136 

23 

F4 

12 

0.52 

23 

F4 

17 

0.136 

23 

F4 

25 

-0.2 

23 

F4 

26 

-0.008 

23 

F4 

28 

0.76 

23 

Ml 

6 

0.088 

23 

Ml 

7 

0.232 

23 

Ml 

9 

-0.008 

23 

Ml 

10 

0.76 

23 

Ml 

11 

0.136 

23 

Ml 

13 

-0.104 

23 

Ml 

18 

0.088 

23 

Ml 

19 

0.712 

23 

Ml 

21 

0.376 

23 

Ml 

23 

0.28 

23 

Ml 

27 

-0.008 

23 

Ml 

29 

0.664 

23 

M3 

2 

0.28 

23 

M3 

3 

0.664 

23 

M3 

5 

0.328 

23 

M3 

14 

0.184 

23 

M3 

15 

0.04 

23 

M3 

16 

0.04 

23 

M3 

20 

0.184 

23 

M3 

22 

-0.152 

23 

M3 

24 

-0.152 

23 

M3 

30 

-0.152 

23 

M4 

1 

0.904 

23 

M4 

4 

0.04 

23 

M4 

8 

0.088 

23 

M4 

12 

0.424 

23 

M4 

17 

-0.008 

23 

M4 

25 

-0.008 

23 

M4 

26 

0.04 

23 

M4 

28 

0.856 

24 

FI 

6 

-0.104 

24 

FI 

7 

-0.056 

24 

FI 

9 

-0.008 

24 

FI 

10 

0.856 

24 

FI 

11 

0.28 

24 

FI 

13 

0.136 

24 

FI 

18 

-0.104 

24 

FI 

19 

0.952 

24 

FI 

21 

0.808 

24 

FI 

23 

0.184 

24 

FI 

27 

-0.008 

24 

FI 

29 

0.904 

24 

F3 

2 

0.76 

24 

F3 

3 

0.808 

24 

F3 

5 

0.376 

24 

F3 

14 

-0.008 

24 

F3 

15 

-0.008 

24 

F3 

16 

-0.104 

24 

F3 

20 

0.616 

24 

F3 

22 

0.088 


Listener 

Talker 

Condition 

RA 

24 

F3 

24 

0.04 

24 

F3 

30 

0.136 

24 

F4 

1 

0.76 

24 

F4 

4 

-0.008 

24 

F4 

8 

0.232 

24 

F4 

12 

0.616 

24 

F4 

17 

0.04 

24 

F4 

25 

0.04 

24 

F4 

26 

0.136 

24 

F4 

28 

1 

24 

Ml 

6 

-0.104 

24 

Ml 

7 

-0.104 

24 

Ml 

9 

0.04 

24 

Ml 

10 

0.808 

24 

Ml 

11 

0.328 

24 


13 


24 


18 


24 


19 


24 


21 


24 

mm 

23 


24 

mm 

27 


24 

— 

29 


24 

miM. 

2 


24 


3 


24 


5 


24 


14 


24 


15 

0.04 

24 


16 


24 

BIB 

20 


24 


22 


24 


24 


24 


30 


24 


1 


24 


4 

0.04 

24 


8 


24 


12 


24 


17 

0.04 

24 


25 


24 


26 


24 


28 

0.76 

25 

FI 

2 

0.52 

25 

FI 

3 

0.76 

25 

FI 

7 


25 

FI 

8 


25 

FI 

9 

0.04 

25 

FI 

10 


25 

FI 

11 


25 

FI 

14 


25 

FI 

16 


25 

FI 

17 


25 

FI 

20 

0.52 | 

25 

FI 

24 


25 

FI 

30 

0.04 

25 

F3 

1 


25 

F3 

5 


25 

F3 

6 


25 

F3 

12 


25 

F3 

13 

0.04 

25 

F3 

18 


25 

F3 

19 


25 

F3 

21 


25 

F3 

22 


25 

F3 

23 


25 

F3 

29 

0.808 

25 

F4 

4 

0.088 


Listener 

Talker 

Condition 

RA 

25 

F4 

15 

-0.008 

25 

F4 

25 

-0.2 

25 

F4 

26 

0.136 

25 

F4 

27 

0.136 

25 

F4 

28 

0.808 

25 

Ml 

2 

0.328 

25 

Ml 

3 

0.76 

25 

Ml 

7 

-0.104 

25 

Ml 

8 

0.28 

25 

Ml 

9 

0.136 

25 

Ml 

10 

0.856 

25 

Ml 

11 

0.136 

25 

Ml 

14 

0.136 

25 

Ml 

16 

-0.152 

25 

Ml 

17 

-0.056 

25 

9ZEB 

20 

0.28 

25 

mm, 

24 


25 

mm 

30 


25 

— 

1 


25 


5 


25 

M3 

6 


25 

M3 

12 


25 

M3 

13 


25 

M3 

18 


25 

M3 

19 


25 

M3 

21 


25 

M3 

22 

0.088 

25 

M3 

23 


25 


29 

0.76 

25 


4 


25 


15 


25 


25 


25 


26 

0.04 

25 


27 


25 


28 


26 

FI 

2 

0.76 

26 

FI 

3 


26 

FI 

7 


26 

FI 

8 


26 

FI 

9 


26 

FI 

10 


26 

FI 

11 


26 

FI 

14 

0.28 

26 

FI 

16 


26 

FI 

17 


26 

FI 

20 


26 

FI 

24 


26 

FI 

30 


26 

F3 

1 

0.76 

26 

F3 

5 


26 

F3 

6 


26 

F3 

12 

0.76 

26 

F3 

13 


26 

F3 

18 


26 

F3 

19 

0.76 

26 

F3 

21 


26 

F3 

22 


26 

F3 

23 


26 

F3 

29 


26 

F4 

4 


26 

F4 

15 


26 

F4 

25 


26 

F4 

26 

0.28 

26 

F4 

27 

0.28 

26 

F4 

28 

0.952 


64 
















Talker 

Condition 

RA 

26 

Ml 

2 

0.424 

26 

Ml 

3 

0.856 

26 

Ml 

7 

-0.008 

26 

Ml 

8 

0.04 

26 

Ml 

9 

0.232 

26 

Ml 

10 

0.76 

26 

Ml 

11 

0.472 

26 

Ml 

14 

0.088 

26 

Ml 

16 

-0.104 

26 

Ml 

17 

-0.056 

26 

Ml 

20 

0.568 

26 

Ml 

24 

0.04 

26 

Ml 

30 

0.136 

26 

M3 

1 

0.856 

26 

M3 

5 

0.568 

26 

M3 

6 

-0.008 

26 

M3 

12 

0.232 

26 

M3 

13 

0.088 

26 

M3 

18 

0.04 

26 

M3 

19 

0.616 

26 

M3 

21 

0.52 

26 

M3 

22 

-0.056 

26 

M3 

23 

0.136 

26 

M3 

29 

0.616 

26 

M4 

4 

0.184 

26 

M4 

15 

0.04 

26 

M4 

25 

-0.008 

26 

M4 

26 

0.328 

26 

M4 

27 

-0.056 

26 

M4 

28 

0.952 

27 

FI 

6 

0.136 

27 

FI 

15 

-0.056 

27 

FI 

18 

-0.104 

27 

FI 

22 

0.088 

27 

FI 

25 

-0.056 

27 

FI 

26 

0.04 

27 

F3 

2 

0.472 

27 

F3 

4 

0.04 

27 

F3 

11 

0.52 

27 

F3 

14 

0.136 

27 

F3 

16 

-0.008 

27 

F3 

27 

0.04 

27 

F3 

28 

0.952 

27 

F3 

30 

0.04 

27 

F4 

1 

0.856 

27 

F4 

3 

0.856 

27 

F4 

5 

0.664 

27 

F4 

7 

0.04 

27 

F4 

8 

0.184 

27 

F4 

9 

-0.104 

27 

F4 

10 

0.952 

27 

F4 

12 

0.616 

27 

F4 

13 

0.136 

27 

F4 

17 

0.088 

27 

F4 

19 

0.664 

27 

F4 

20 

0.472 

27 

F4 

21 

0.568 

27 

F4 

23 

0.04 

27 

F4 

24 

-0.056 

27 

F4 

29 

0.664 

27 

Ml 

6 

-0.104 

27 

Ml 

15 

0.04 

27 

Ml 

18 

0.04 

27 

Ml 

22 

0.088 

27 

Ml 

25 

-0.104 


Listener 

Talker 

Condition 


27 

Ml 

26 


27 

M3 

2 


27 

M3 

4 


27 

M3 

11 

0.088 

27 

M3 

14 


27 

M3 

16 


27 

M3 

27 


27 

M3 

28 


27 

M3 

30 


27 

M4 

1 


27 

M4 

3 


27 

M4 

5 


27 

M4 

7 


27 

M4 

8 


27 

M4 

9 


27 

M4 

10 


27 

M4 

12 


27 

M4 

13 


27 

M4 

17 


27 

M4 

19 


27 

M4 

20 


27 

M4 

21 


27 

M4 

23 


27 

M4 

24 


27 

M4 

29 


28 

FI 

6 


28 

FI 

15 


28 

FI 

18 


28 

FI 

22 


28 

FI 

25 

0.04 

28 

FI 

26 

0.04 

28 

F3 

2 


28 

F3 

4 


28 

F3 

11 

0.28 

28 

F3 

14 


28 

F3 

16 

0.28 

28 

F3 

27 


28 

F3 

28 


28 

F3 

30 


28 

F4 

1 


28 

F4 

3 


28 

F4 

5 

0.52 

28 

F4 

7 

0.04 

28 

F4 

8 

0.04 

28 

F4 

9 

0.04 

28 

F4 

10 


28 

F4 

12 


28 

F4 

13 

0.04 

28 

F4 

17 


28 

F4 

19 

0.76 

28 

F4 

20 

0.52 

28 

F4 

21 


28 

F4 

23 

0.52 

28 

F4 

24 


28 

F4 

29 


28 

— 

6 


28 

an 

15 


28 

an 

18 


28 

™ 

22 


28 

an 

25 


28 

an 

26 


28 

an 

2 


28 

an 

4 


28 

M3 

11 

0.136 

28 

M3 

14 

-0.008 


Listener 

Talker 

Condition 

RA 

28 

an 

16 


28 


27 

0.04 

28 


28 


28 


30 


28 


1 


28 


3 


28 


5 


28 


7 

0.04 

28 


8 


28 


9 


28 


10 


28 


12 


28 


13 


28 


17 

0.04 

28 


19 


28 


20 


28 


21 


28 


23 


28 


24 


28 


29 


29 

FI 

1 


29 

FI 

4 


29 

FI 

5 

0.52 

29 

FI 

12 


29 

FI 

13 


29 

FI 

19 

1 

29 

FI 

21 


29 

FI 

23 

0.52 

29 

FI 

27 


29 

FI 

28 


29 

FI 

29 


29 

F3 

3 


29 

F3 

7 


29 

F3 

8 


29 

F3 

9 


29 

F3 

10 


29 

F3 

15 


29 

F3 

17 

ain^i 

29 

F3 

20 


29 

F3 

24 


29 

F3 

25 

0.04 

29 

F3 

26 

unwind 

29 

F4 

2 


29 

F4 

6 


29 

F4 

11 


29 

F4 

14 

0.04 

29 

F4 

16 


29 

F4 

18 


29 

F4 

22 


29 

F4 

30 

0.04 

29 


1 


29 


4 


29 


5 


29 


12 


29 

Ml 

13 


29 

an 

19 


29 

an 

21 

0.52 

29 

ran 

23 


29 

an 

27 


29 

an 

28 


29 

ran 

29 

0.76 

29 

SB 

3 


29 

an; 

7 

0.088 

29 

M3 

8 

0.088 

29 

M3 

9 

0.088 


65 


















RA 

29 


10 


29 


15 


29 


17 


29 


20 


29 


24 

-0.2 

29 


25 


29 


26 


29 


2 


29 


6 


29 


11 


29 


14 


29 


16 


29 


18 

0.04 

29 


22 


29 


30 



FI 

1 



FI 

4 



FI 

5 



FI 

12 


30 

FI 

13 

0.28 

30 

FI 

19 

0.904 

30 

FI 

21 

0.616 

30 

FI 

23 

0.28 

30 

FI 

27 

0.184 

30 

FI 

28 

0.856 





RA 

30 

FI 

29 


30 

F3 

3 


30 

F3 

7 


30 

F3 

8 

0.28 

30 

F3 

9 


30 

F3 

10 


30 

F3 

15 


30 

F3 

17 

0.28 

30 

F3 

20 


30 

F3 

24 


30 

F3 

25 


30 

F3 

26 


30 

F4 

2 


30 

F4 

6 


30 

F4 

11 


30 

F4 

14 


30 

F4 

16 


30 

F4 

18 


30 

F4 

22 


30 

F4 

30 

0.088 

30 

Ml 

1 

0.952 

30 

Ml 

4 

0.136 

30 

Ml 

5 

0.568 

30 

Ml 

12 

0.376 

30 

Ml 

13 

-0.008 


Listener 

Talker 

Condition 

RA 

30 

— 

19 


30 

mm 

21 


30 

mm 

23 


30 

—i 

27 


30 

mm 

28 


30 

mm 

29 

0.76 

30 

S3! 

3 

0.52 

30 


7 


30 


8 

0.04 

30 


9 


30 


10 


30 


15 

0.04 

30 


17 


30 


20 


30 


24 


30 

S3 

25 


30 


26 


30 


2 

0.52 

30 


6 


30 

M4 

11 

0.28 

30 

M4 

14 

-0.008 

30 

M4 

16 

0.04 

30 

M4 

18 

-0.104 

30 

M4 

22 

0.232 

30 

M4 

30 

0.04 


66 








APPENDIX C: SOURCE SPEECH RECORDING INFORMATION 


Some speech material in this test was recorded in the laboratory at the Institute for 
Telecommunication Sciences in Boulder, Colorado. Recordings were perfonned in an NC-35- 
rated sound isolation chamber according to the following. 

The microphone used in the source recording was a Shure Beta 53A microphone sampled at 48 
kHz/16 bit on a Windows-based computer using commercially available software. Active signal 
level was nonnalized to -28 dB below overload using the ITU-T Recommendation P.56 
voltmeter software [6] [11]. 

The speech material spoken by the talkers was the word list defined in the MRT description of 
[3] in the carrier sentence, “Please select the word ....” 
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APPENDIX D: VOICE CODER SHORTHAND NOTATION 


Table D-l. Voice Coder Shorthand Notation Used by TIA 


New 

Name 

Modulation 

Name in 
2003 

Gross b/s 

Net b/s 

FEC 

Back 

Ground 

Noise 

Tones 

QFA 

QPSK-c 

Baseline 

7200 

4400 

Hard Dec 



QHA 

QPSK-c 

HR 

3600 

2450 

Soft Dec 



FFB 

F4FM 

— 

6300 

4400 

Soft Dec 

X 

X 

FHB 

F4FM 

— 

6300 

2450 

Soft Dec 

X 

X 

QDB 

QPSK-c 

— 

7200 / 3600 

4400 / 2450 

Soft Dec 

X 

X 

QFB 

QPSK-c 

EFR 

7200 

4400 

Soft Dec 

X 

X 

QHB 

QPSK-c 

EHR 

3600 

2450 

Soft Dec 

X 

X 

QHC 

QPSK-c 

EHRS 

3300 

2250 

Soft Dec 

X 

X 


First Character Modulation 

Q QPSK-c modulation, includes C4FM and CQPSK 

F F4FM modulation 


Second Character 
F 
H 
D 


Full Rate/Half Rate/Dual Rate 
Full Rate 
Half Rate 

Dual Rate (Full and Half) 


Third Character 
A 
B 
C 


Suffix Enhancement/Bit Stealing 
Basic, not enhanced 
Enhanced 

Enhanced with bit stealing 
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APPENDIX E: LISTENING LABORATORY CONFIGURATION 


Two test chambers were set up to meet the standards set forth in Sections 8.10.4.10 - 8.10.4.15 of 
[9]. The physical layout of the chambers can be seen in Figure E-l. The loudspeaker carrying 
the speech signal sat on a table and was placed equidistant from the chamber side walls, on the 
edge of the table nearest the listener. The listening position was equidistant from the chamber 
side walls, and 150 cm away from the speech loudspeaker (in analogy to the talker-listener 
distance specified in 8.10.4.10). The two loudspeakers on either side of the table were used to 
produce the pink noise (allowed by 8.10.4.14). The loudspeakers were pointed toward the 
“back” of the room, and were not pointed directly at the listener, thus fulfilling 8.10.4.12. The 
combination of using two loudspeakers to produce the pink noise and the distance from the 
loudspeakers to the listening position created a quasi-unifonn field of sound, thus satisfying 
8.10.4.13. 

In order to generate a field of “broadband pink noise” in compliance with section 8.10.4.11 of 
[9], a Gold Line Model PN2 Pink Noise Generator (PNG) was used. The PNG was modified to 
accept an external power source so any possible effect of a non-constant battery voltage could be 
avoided. The output of the PNG was fed into a General Radio Model 1952 Universal Filter, 
which was tuned to have a bandpass characteristic for the interval between 400 Hz and 4 kHz. 

The signal was then split into two signals in order to supply two different test chambers. After 
splitting, each signal passed through a mixer, equalizer, power amplifier and was finally 
delivered to the pair of loudspeakers located in a chamber. A Gold Line Model DSP30RM 
realtime spectrum analyzer was used to analyze the resulting acoustic pink noise spectrum in 
each chamber. Using information from the spectrum analyzer, the equalizers were tuned to 
achieve the best possible response for each test chamber. 

Section 8.10.4.11 specifies that the pink noise generated should have a tolerance of 6 dB per 
octave band in the range of frequencies between 400 Hz and 4 kHz. In this range there are 
eleven third-octave bands. This means that only three full octave bands can be measured in this 
range, yet three octave bands do not fully cover the region of interest. However, it is possible to 
analyze two different sets of three full octave bands by grouping third-octave bands as shown in 
Tables E-l and E-2. These two sets of bands are called octave band set A (OBSA) and octave 
band set B (OBSB). 

After equalization was performed for each experimental chamber, the pink noise in each 
chamber was recorded and analyzed. A third octave analysis was performed in MATLAB. 
Energy in OBSA and OBSB were calculated by calculating the sum of the three third-octave 
bands included in their respective octave band. The relative noise power results for OBSA and 
OBSB are shown in Table E-l and Table E-2 respectively for each chamber. For each chamber 
the 0 dB reference point is the midpoint between the highest and lowest measured octave band 
noise power. These tables show that 8.10.4.11 is satisfied in all cases. 

A Briiel and Kjser Model 2250 sound level meter was used to verify that the noise level met 
specifications. The noise level in Chamber 1 was measured to be 69.4 dBA, and the noise level 
in Chamber 2 was measured to be 69.7 dBA, fulfilling 8.10.4.15. 
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Figure E-l. Listening laboratory layout. 


Table E-l. Relative RMS Noise Power per Octave in Two Chambers Using OBSA 


Third Octave Bands 

Chamber 1 

Chamber 2 

400, 500, 630 Hz 

+4.1 dB 

+4.0 dB 

800,1000, 1250 Hz 

+1.6 dB 

+2.9 dB 

1600, 2000, 2500 Hz 

-0.5 dB 

+1.3 dB 
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Table E-2. Relative RMS Noise Power per Octave in Two Chambers Using OBSB 


OBSB 

Chamber 1 (in dB RMS) 

Chamber 2 (in dB RMS) 

630, 800, 1000 Hz 

+2.1 

+3.0 

1250, 1600, 2000 Hz 

+0.2 

+2.4 

2500, 3150, 4000 Hz 

-4.2 

-4 


Speech was generated in the two chambers using Fostex 630IB loudspeakers. The speech signal 
originates in MATLAB, propagates through the PC’s sound card, then to a mixer, and finally to 
the loudspeaker. When the signal path was active but no signal was being sent to the speakers, 
the noise level in Chamber 1 fell to 22.3 dBA, and the level in Chamber 2 fell to 20.3 dBA. 
When speech was active in Chamber 1, speech levels ranged from 67-88 dBA, with undistorted, 
noise-free speech registering around 82 dBA. Similarly, active speech levels in Chamber 2 
ranged from 70-86 dBA, and undistorted, noise-free speech registered around 82 dBA. 
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